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ABSTRACT 

This report details the design, development, and 
implementation of computer fsoftware to support the cost-effective 
production of computer assisted irstruction (CAI) within the context 
of the Advanced Instructional system fAISI located at Lowry Air Force 
Base. The report supplements the ccmpoter managed Air Force technical 
training that is currently supported by AIS, giving the Air Force a 
full function computer based instructional system. In describing the 
interactive authoring editor, presentation program, data collection, 
and data print software ccmpcnents of the CAI system, this report 
indicates that the editor simplifies the authoring task by (a) 
eliminating the need for the author tc use a computer language, (b) 
structuring the task, (c) providing computer-aided input, and (d) 
providing extensive formatting and editing capabilities. The software 
also provides conditional and unconditional branching that can be 
specified from the editor. (Author) 
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Objective 

The overall objective of this project was to design, develop, 
implement, and evaluate an authoring system which would provide a 

r e^^ective production of computer-assisted Instruction 
JW) materia H tor use In the context of computer-managed Air Force 

tn^JIL^.^jL!*'?^?^"?- 11;?.?'^?*^^^^^ application was the Advanced 

r^^*^^5*^ ^y^^^^ (AIS) located at Lowry Air Force Base, Colorado. 
i^il^^ software developed was to be Integrated into this system. The 
project work was conducted through two parallel efforts which are 
described in Parts I and II of this two-part report. The first of these 
efforts, described here in Part I of the report, addressed the develoo- 
raent of computer software to facilitate the authoring, presentation, 
and evaluation of CAI materials. The second effort, described in P^rt 
U, concerned the definition qf a procedural model for CAI production 
and evaluation of the complete authoring system. 

Approdcr. 

rJi^^i'il^^^lJ^llV^^^^ ""^^^ analysis of the probable functions 
OT LAI Within the AIS. review of prior approaches to supporting CAI 
materials development, examination of previous military CAI development 
experiences, analysis of the characteristics (training, prior experience 
and work environment) of the Air Training Command (ATC) personnel who 
would be developing CAI materials, and re-exaul nation of the available 
Alb software and operational experience with this software. A major con- 
clusion resulting from these analyses was that there are a number of 
factors in the military technical training environment which are incom- 
patible with the typical approach to CAI production (authoring) and that 
prior attempts to utilize CAI in this environment had not taken these 
factors into sufficient consideration. Typically, it appeared that too 
much was expected of the CAI authors. It was decided, therefore, that 
It was preferable to adapt the authoring system to the existing environ- 
nwnt rather than expect the environment to change to meet the require- 
ments of the system, even when this approach limited the sophistication 
ot the CAI materials which could be produced. For example, it was con- 
sidered preferable to avoid author use of a programnlng language even 
though this would limit the author's flexibility. The AIS employs 
editors which engage the user in an interactive, English language dialog 
to control tne system's data oase. Experience with these interactive 
editors suggested that they could provide a model for the author/computer 
interface. It was also concluded that the system should structure the 
author s task, promote principles of good instructional design, require 
very little training for its use. and provide aids for managing a 
materials development project. 

The neart of the software developed is an interactive CAI Authoring 
Lditor. The Editor structures the author's task while providing options 



as to CAI proiuntdtiun strvite'jy detail s» As defined by the Editor* a 
CAI module is divided into objectives. The system is frdn>e oriented and 
eacn objective can contain up to 100 frames. Tnree classes of frame 
types are supported: textual content frames j question frames and special 
•Jurpose frames. The author enters the frame content in exactly the 
format in wnicn it will be seen by the student. All formatting of 
^luestion frames is done automatically and the autnor is prompted to 
supply d feedback statement for each alternative or constructed response 
and a ;:ronpt statement for each attempt at answering the question. To 
individualize instruction, the author can define branches from any frame 
to any otner frame. The decision of whether or not to branch can be 
based on a specified number of questions being answered correctly or 
incorrectly* on a set of frames navJng or not having been presented* or 
on a specific student response. Branching logic is entered In a highly 
pronpteo* multiple choice format, and the result is then displayed in 
tnul ish. 

cAI materials are delivered by a CAI Presentation Program. A 
standard presentation proqran was first written for CAI as an alternative 
instructional n\odule. Two modified versions of this program provide 
review anu remediation over the specific objectives with which the 
student nad problems. The presentation programs contain student per- 
foniance data collection routines which can be turned on or off by the 
auti»or. Standard data analysis reports are requested by means of a 
second interactive editor which prompts the user as to the options avail- 
able. 

A variety of printouts of program content and logic are provided, 
ludultr content can also be printed out in a format suitable for student 
jSc as d programned text. 

L valuation Procedures and Results 

The software developed was evaluated as part of the total authoring 
syste»t tnrough the production and implementation of a set of CAI lessons 
in one of tne courses supported by AIS and by training a number of course 
personnel in tne use of the authoring system. The evaluation procedures 
ana results are described in detail in Part II of this report and are 
only suunarized here, in Part I, 

Under tne materials production effort, CAI modules were developed 
for six lessons in the AIS Weapons f4echan1c course. None of the three 
members of t^e authoring team had prior CAI experience; although, all 
were experienced technical training authors. Approximately 2200 man 
hours v<ere required for development of the six modules. The modules 
accounted for a total of approximately 2b Plan of Instruction (POI) 
nours and resulted in an average student contact time of Id. 7 hours. 
Thus, development efforts required an average cf 88 man hours per POI 
hour and llo man hours per student contact nour. This compares very 
favorably with the figures of 222 and 246 man hours per contact hour 
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reported by Himwich (1977) for military technical training CAI. 

HurinnM%?I^J?^^!;"^^°''^ ''^^^ trained in use of the Authoring Editor 
nrnnr^nJiLc T^^! "^^^ sessions. None of the trainees were computer 
fnZ^ fr.Li'' P^^o^ ^-'AI development experience. There was no 

tnZll II I i ' ^V^^ ^"5^'°"' S^'ven a CAI Author's Handbook 

hLo?! manual, eacl, trainee used the Authoring Editor to 

aZlliLl .tri Contractor personnel were available to answer 

SeHnd Lrh hlHT'' trainees' work. At the end of the training 
inn^?r;n?f 1 developed a module through the stage of revision follow- 
Ind ?hP^l^Hn«c'"^ ^^'^ trainees asked relatively few questions 

fitu w!?? °^ generally good quality and capitalized 

fairly well on the capabilities of CAI. The trainees were quite 
satisfied witn the Editor and all expressed an interest in implementing 
LAI 1(1 their courses, 

Concl usions 

The approach taken to facilitating CAI development appears very 
promising. Experience to date, while limited, has demonstrated that con- 
tractor 'personnel were able to produce effective CAI with a level of 
ertort chat is comparable to the effort required to produce paper and 
penci materials. ATC personnel were able to learn to use the Authoring 
tdito and to produce CAI materials within a very short period of time. 
Thesw trainees expressed highly favorable attitudes about the approach 
and found no serious faults with the Editor. 

The Kidjor factors which contribute to simplifying the authoring 
taSK are probably elimination of any need for the author to use a com- 
puter language and the extent to which the task is structured. The human 
ermineered, computer-aided input, formatting, and editing capabilities 
provided by the Editor are undoubtedly also important. 

The authoring system is ready for use by ATC instructional develop- 
ment personnel in Us current form but it should not necessarily be 
considoreu to be a finished product. There are a number of features 
which could be added to increase its utility. As the system is used, it 
can be expanded and refined on the basis of accuMulatinq experience to 
uecoiiie a ;)ov/erful tool for instructional development. 
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Conputer-based instruction has the pot^Tntial of achievl'^g signif- 
icant savings in costs related to .41r Force technical training,, number 
of steps toward realizing thlj. ptTtential have already teen taken, e.g., 
use of the PLATO system >ftd" development of the Advanced Instructional 
System (AIS). The fortner is an application of computer-assisted in- 
struction (CAI> while the latter represents a comprehensive computer- 
based instructional (CBI) system supporting both computer-managed in- 
struction (CMI) and CAI. 

CMI can be defined as a situation In which the majority of the 
stuoent's Instructional activities are completed.jttff-line. The 
computer's role is tnat of evaluator, diagnostician, prescriber, and 
manager of instructional events, 'a CAr," by contrast, all of the stu- 
dent's instructional activities are conducted on-line, at an inter- 
active computer terminal. CMI can be characterized as being extensive, 
managlno Instruction for a large number of st». .ants throughout a large 
body of course content. CAI, on the other hand is typically intensive, 
concentrating on detailed and highly interactive Instruction for a 
limited segment Of course content and a relatively small number of 
students, txtensive application of CAI has, to date, been limited by its 
nign cost, in terms of both materials production and terminal costs, 
and by tne limited utility of insular segments of individualized instruc- 
tion in a group-paced environment. 

The work reported herein addresses the concept of a comprehensive 
Instructional system in which CAI Is embedded in the context of CMI, 
Such an integrated system has a number of distinct advantages. Student 
pacing, being individualized, is compatible with efficient use of CAI. 
Student performance on CAI lessons can be recorded directly by the CHI 
system. Most importantly, the extensive student performance records 
maintained as part of CMI can be readily accessed to provide truly 
individualized CAI, when and where it is most needed. 

The relatively expensive CAI must, however, be employed judiciously 
in the CMI context to be cost effective, If one assumes that the off- 
line instructional materials being managed by CMI are reasonably 
effective, the use of CAI Is only approprlMe when (a) the concepts or 
facts to be taudht are uniquely difficuif'and existing instructional 
materials and methods are Inadequate for a large proportion of the 
students, or (b) logistical problems can be resolved through the use of 
CAI. The cost of producing and delivering CAI for specific applications 
is, of course, also an important consideration. 

Project Purpose 

There are two basic reasons why CAI production costs remain high. 
With few exceptions, the programning languages and production methods 




12 



employed require extensive and very detailed effort. Secondly 
restively si^phlstlcated Instructional design skills are required as 
wen as, in n^st cases, computer progranming skills. 

tralnlno'envi^Tnlllln^ ^^J^i' ^^^^'^ '"^l^tary technical 

traininq environment, CAI development problems can be alleviated and 

f ^^"5^ structuring the authoHng asf anS pro- 
" 2ih2r no ^ll ^''^H^'' '"^'^"^"9 process! ?he Um 
o^nArAMnn' . 1 ''^^'^ ^'''^ ''®P°'^^» '^^^^'•s to the process of 
Sle Surh't- i:^^"?^* and revising an individualized, interactive CAI 
module. Such a mooule consists of Information to be assimilated by the 
student, questions and/or practice exercises, decision rules for 
I?urt2;"n!ir"^ ^"^"'^^ "^^"""^ 0^ the instruction on the basis of 
maslerv orthl^^lH'i ^^^'^^^^^ or exercises to measure the siSdent's 
mastery of the module»s objectives. It was hypothesized that aooroDrlate 

ment rn:.if h]^"^?!'^"'!'^*^^'^^ ^° CAI authoring In threnSl^j! 

Td eva?uI?ion'''f ^w'""^ substantially reduce the cost of CAI develop^nt 
ana eva uation. It was also thought that skill level reaulrements anrf 
production time could be further reduced if the dlsfgn^deieWnt 
Leci? c°ornr'/''^'^°:: ''f' ^^'^hly structured'Jhrough X use of 
P^oce^^es ' '''^''^^^ ''^'^^ supported and enforced these 

.unonIr/h!ntr^ developed under this project are intended to 

j^^^TLr, ^.J'^^'fj^^^^^^ of tutorial and drill and practice tasks. 
Tney would not be as useful, for example. In developinq CAI to simulatP 

?exJTrien?L'ir'"'^*H V"'''"''* authoring ^^'IJeVSLeUrls'' 
^r^^^te^ opposed to supporting extensive production of comouter- 
generated qrapnics. Unless sophisticated qraphics production software 
Tro ectt'^Jh^f"' ? ^? t beyond'tSe scope of this 

exoin^iv; Li;r??f S'?" '"k ^^^^^e'-y 0^ computer graphics Is quite 
expensive. Additionally, when use of supplementary hard copy visuals 
le.g scnenatics and photographs) is considered, there are relatively 
few tutorial or drill-and-practice applications in which computer gen^ 
erated qrapnics can be shown to be cost effective. 

Project co ntext: Tne Advanced Instructional System 

• Tne autnorinq support software was designed for application In the 

is" l' or?nu' ColSraSo ?he aJs 

^frnrnL ?ruM^ T^^^^ ^'"^^ ^ ^^^^^ ^^ale computer-based in- 

struction (CBI) system supported by hardwa*-e and software desioned to 

tiwinlcc'^'n '^^l'"*^ system was designed to ?^rove ?he ef?ec- 

n "^^f efficiency of Air Force techhical training and to Providian 

^'"^^^'^ assessing innovations in Instru?^ onal 
tecnnology The system supports four technical training courses repre- 
on?^?^!^/ ""^"^^ cognitive and performance skills required by 

An^n!f.^It. 1.''!^^^'''''""^^* An adaptive Instructional decision model 
aZnZl ^^ff-o^-*hj:^'-\ computer hardwa. 2. software, statistical method- 
oioaies ano instructional procedures to provide instructional management 
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AIS Course Structure . Each AIS course is divided into "blocks" of 
instruction wntcb ma> require from 1 to 10 days to complete. Each block 
contains a number of lessons and a comprehensive, end-of-block test. 
Within a block, lessons are arranged in a hierarchy based on their pre- 
requisite relationships. A typical hierarchy resembles a set of parallel 
chains diverging and converging on certain pivotal lessons, and a student 
may alternately work on lessons in two or more parallel chains. 

The basic unit of instruction is the lesson. Each lesson consists 
of a set of objectives, two or more parallel forms of a crtterjon 
referenced test, a criterion defining adequate mastery on the test, and, 
typically a self-test by which students can evaluate their under- 
standing of the lesson before taking the criterion test. 

A lesson's instruction is provided by one or more modules, each of 
wnich teaches tne complete lesson content. When two or more modules are 
present, they represent alternative instructional treatments. Depend- 
ing on the lesson content and the nature Of the treatment, a module may 
be a progranined text, an elaborated technical order, an audio-visual 
presentation, or given the results of this project, an interactive CAI 
session. 

An AIS Student Scenario. A student's first experience with AIS is 
to complete a preassessment battery consisting of a number of scales 
which assess cognitive and affective factors considered to be predictive 
of students' performance in the course. The student then requests an 
initial assignment by submitting a Forward-Going Assignment request at 
a management terminal, which consists of an optical scanner and medium 
speed printer. At this point, the student is enrolled in the course but 
has not yet entered a block containing actual course content. First, 
tnerefore, the system selects the block in which the student is to start 
work. Since the student has not yet completed any course work, only 
blocks which have no prerequisites are considered. If there is more 
than one such block, the one containing the fewest students relative to 
the desired number in that block is selected. The student is then 
assigned to an appropriate learning center and home carrel and to a 
specific lesson, module, and criterion test. 

Lesson assignment decisions are made jointly by two major 
components of the System— the Adapter and the Resource Allocation Model, 
The Adapter attempts to select, for each assignable lesson, the one 
module that is most appropriate for that student. This decision can 
be based on a variety of rules, e.g., select the module which the 
student is predicted to complete In the shortest time (assuming the 
student is also predicted to pass the criterion test). Each alternative 
module is given a weight indicating Its relative preference. The 
Resource Allocation Model assigns preference weights to modules on the 
basis of minimizing the Impact of the assignment on the availability of 
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instructional resources. The final lesson and module selection Is based 
on a compromise between the two sets of preference weights. The form of 
the criterion test is chosen at random. 

The student, after receiving the first assignment printout (called 
a Student Status Report) at the management terminal, reports to the 
learning center instructor, obtains the instructional resources required 
for the assigned module, and begins work, normally at a home carrel. 

After studying the lesson materials, the student completes a 
multiple-choice self- test and reviews the material pertaining to any 
questions answered incorrectly. The student then completes the lesson 
criterion test and submits the test form to a management terminal. The 
resulting Student Status Report details the student's performance on 
the criterion test (percentage total score, items missed, objectives 
failed, and pass/fail decision) and the next assignment. If the test 
criterion was not met, the student is reassigned the same lesson and an 
alternate form of the test. Otherwise, the lesson, module, and test 
selection procedures are repeated, and the student is assigned a new 
lesson. 

If the student's assignment is a CAI module, there is only a slight 
variation In tnese procedures. Tne function of the self- test is 
assumed by questions embedded in the CAI presentation. The criterion 
test is also administered on-line and the results submitted auto- 
matically to the CMI system. Tne Student Status Report is displayed 
on the terminal and a printed copy of the report is also available from 
the manaqenent terminal. 

When the student has completed all content lessons in the block, 
a dlock Review lesson is assigned. Following review, the student is 
randomly assigned one of the alternate forms of the block test. While 
lesson tests can be viewed as diagnostic tools, end-of-block tests 
serve a certification function. Tnat is, since there is no end-of 
course test, block test performance serves as the basis for certifying 
mastery of the objectives contained in the block. A student not meeting 
the block test criterion is reassigned to the block in a status whereby 
assignments are nade by the instructor rather than by the system. If 
the block decision is "Go," the block selection logic is repeated, and 
the student is assigned to the next block of study. The student's con- 
tinued progress through the course is essentially a repetition of these 
events. 



15 

11 



.II AUTHORINu SYSTC'^ DESIGN CUNSIDE^ATIONS 



uesiqn of a cost-effective CaI component for the AIS beaan with an 
analysis of . tne AIS environment: the appropriate role of Ikl within a 
CH s/stei: su,)i>oninn nilitary technical traininq; the characteristics 
of tnt» personnel wno, it was anticipated, woi-ld be developinq CAI 
TKiterials; and tne software tools currently available within the AIS. 
Lessons learneu from prior approaches to CAI development were also 
considered. 

Tne Kole of CAI wunin the AIS 

Klrst» it must be recognized that tne prototype AIS is primarily a 
compute r-manaqed instructional system. This is not to imply, however, 
that the system was not designed to accomnodate CAI. The System 
lannuaoe, CAMIL (Computer Assisted/Managed Instructional Language), was 
specifically developed to suppnn DOth CAI and CHI. Rather, within the 
context of tne AIS, CAI was seen as one of several possible media avail- 
able for instructional purposes. Manaqement and monitoring of students* 
Dronress throuin a course, assionnent to soecific instructional treat- 
ments, and evaluation of instructional effectiveness are all supported 
by the QM component of the system. 

alv^n the nature of the AIS form of CMI, defining the role of CAI 
within tnis structure was relatively straightforward. Recall that an 
AIS course ^s divided into blocks of instruction whicn conclude with 
certification tests. Blocks are divided into lessons, and each lesson 
is supported by one or more modules, each of which addresses all of the 
lesson's oojectives. Wnen more than one wdule Is available for a 
lesson, tn;.'y are treated as alternative Instructional treatments for 
that lesson. Tne Adaptive Model assures that a student is not assigned 
a lesson until all prerequisites have been completed. Assignment of a 
specific nodule is also a CMI function. Thus. CAI was seen as providing 
one of two or more alternative Instructional treatments for teaching 
a lesson, and as such, CAI materials would be packaged anu assigned as 
:»o(iules. Student terminals required for CAI would b2 treated as 
instructional resources managed and assigned by the Adaptive Model. 

It was assumed that if a student was assigned and .-completed a CAI 
module, it would be desirable that the lesson test also be administered 
on-line rather tnan via a management terminal interaction. While the 
AIS does support an on-line, computer-assisted testing (CAT) capability, 
it was designed primarily for block tests and was not totally suited 
for administration of lesson tests. Therefore, a' lesson-level testing 
capability was incorporated into the CAI component. 

The next major question concerned the types of applications for 
which these relatively expensive modules woi»ld be most effective. While 
the use of CAI for nonral first-pass Instruction was one obvious answer, 
it was hypothesized that the branching capabilities and moment-to-moment 



control over student behaviors afforded by CAI would be particularly 
useful for the functions of review and remediation. 

As AIS courses are currently structured, each block ends with a 
review lessen which is assigned immediately prior to the block test. No 
specific instructional modules had been developed tc support these 
lessons. Rather, it was intended that students would use this assign- 
ment to review those block objectives on which they had encountered 
problems. The CMI system provides a "Block Report" which flags the 
objectives wnich the student failed on the first attempt at the lesson 
test. Instructional activities during this period are, however, 
determined primarily by the student and the instructor, and student 
performance data indicated that this time was often not being used 
effectively. CAI was seen as an excellent way of remedying this 
situation. 

A procedure was envisioned in which, when a student was being 
assigned to block review, the Adaptive Model would assess the student's 
prior performance in the block and, if performance was found to be 
marginal, assign a CAI module. The CAI nodule would, in turn, determine 
the specific objectives on which the student had encountered problems, 
review the student on these objectives, administer and evaluate 
objective-level diagnostic tests, provide further remediation as 
necessary, and issue a Status Report suggesting further review or 
assignment to the block test as appropriate. 

Block remediation presented a similar situation. A student who 
fails a block test is placed in "block remediation" mode. In this mode, 
the System accepts any lesson tests input by the student or instructor 
input of a second attempt block test. The System does not. nowever. 
make any specific assignments. This role is delegated to the 
instructor. To guide the instructor in making appropriate assignments, 
the objectives wnich the student failed on the block test are listed 
'on tne Student Status Report printed when the block test is scored. 
Again, student performance data suggested that this remedial time could 
be employed much more effectively. 

CAI block rtmediation modules were envisioned which would differ 
only slightly from the review noduies. Student assignments to CAI 
remediation would be made by the instructor, and selection of specific 
objectives for remediation would be based on the student's block test 
performance rather than on performance in the block. 

Although block review and remediation were seen as two prime 
targets for CAI, it was assumed that CAI would also be used for 
alternative modules for first pass instruction. While CAI might 
occasionally be used for the first module developed for a lesson, as 
long as CAI costs remain high, it was expected that it would more often 
be used as an alternative treatment designed to remedy specific problems 
detected in existing modules. 



Anticipated Characteristics of CAI Development Personnel 



In considerinq the characteristics of the personnel who would be 
developinn CAI Ttodules. it was first thought that a team approach, such 
as advocated by Sunderson (1973), would be appropriate. Such an approach 
would soecify differing roles for (at least) subject nwitter experts, in- 
structional tecnnoloqists, and program coders. However, further analysis 
of the military technical training environment strongly suggested that, 
wrile desirable, the team approach was very likely to encounter serious 
problens in practice. 

liurinq the 5 years in which the AIS has been operational, there 
have,been repeated efforts to define specific roles for the specialists 
wno are so important to effective operation of CMI (e.g., materials 
writers, evaluators. and data case manaqers). To date, these efforts 
nave nau only limited success. It was concluded, therefore, that the 
CAI autnorinq system should be structured so as to allow a team 
approach to CAI development but should not be dependent on it. 

For the irrmediate future, at least, it was reasonable to expect 
that the personnel wno would be developing, evaluating, and revising CAI 
:ndterials woulti, for the most part, be classroom instructors. It was 
assut;ie(i that sucn authors would typically be expert in their subject 
•natter area, would have limited training or experience in instructional 
systems Gesign, would have no prior exposure to CAI, and would have no 
computer programming experience. Additional pertinent characteristics 
included a relatively nigh turnover rate for military instructors, 
little or no opportunity for formal training, and limited, fragmented 
periods of availability (e.g.. 60 to 90 minutes following a normal 
instructional day). 

An obvious problem raised by this profile of the typical CAI author 
is the lack of computer programming expertise and the strong indication 
that attempting to train relatively transient authors to program would 
not be practical. Therefore, an approach was sought which would 
eliminate, or at least substantially reduce, the need for computer 
programing on the part of the author. 

Another implication of the analysis of authoring-personnel 
characteristics was a need for procedures which would structure the 
authoring task. At the same time. It was recognized that excessive 
structuring could be perceived as being undesirable, even offensive, 
and could consequently detract from the authors* motivation. It was 
reasoned that they would be more amenable to task structuring once CAI 
was establlsned and authoring problems had been recognized. It was con- 
eluded that the authoring system should provide a degree of task, 
structuring necessary to prevent a novice author from becoming hope- 
lessly bewildered and should also provide for Implementing more 
extensive structuring In the future. 
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The high turnover rate of military personneT had the obvious 
implication that the authoring system and procedures should be such 
that a novice author could quickly be brought to a productive level of 
proficiency. Further, it was apparent that it would often be necessary 
for new autnors to complete CAI modules that had been designed and 
partially develrped by others. 

Coupled with tne high turnover rate wes the expectation that there 
would be 11*-tle or no opportunity for formal training in either the 
mechanics of developing CAI materials or the instructional principles 
underlying their design. It was assumed that most training would have 
to be conducted on the job. Thus, self- instructional methods, with 
some minimal assistance from experienced authors, appeared necessary. 
It was also desirable that the software tools themselves be self- 
documenting and self-instructional. 

Since it was anticipated that authors would often have to work in 
short segmented oeriods or in an environment subject to frequent inter- 
ruptions, it was desirable that they have rapid access to the portion 
of the module on which they were working, that any work accomplished be 
captured Irmnediately rather than requiring a lengthy storage process, 
and that the current status of work accomplished be clearly summarized. 

Finally, experience gained from various DOD projects indicates that 
management of instructional materials development is an area that has 
been generally problem prone because of its complex, unstructured nature. 
Few effective management procedures have been successfully Implemented. 
Consequently, little monitoring of individual authors during materials 
development nas ueen possible, especially with respect to productivity 
and quality control, thus, there was a need for software tools that 
would facilitate management of the development process. As was tne 
case for structuring the authoring task itself, it was necessary that 
these management tools be fairly open-ended and allow for further 
development as nanagement procedures evolved. The review of completed 
materials by other subject matter experts was known to be particularly 
time consuminq. Software tools which would structure and accelerate 
this stage were considered especially important. 

CAI materials evaluation haj potential -for problems which were at 
least equal in severity to those of authoring per se. It was assumed 
that tne CAI authors would have primary responsibility for formative 
evaluation of their own materials but would have little, If any, prior 
evaluation experience. As a result, there was a need to structure 
student performance data collection, retrie/al, and reporting. In this 
case. It was thought that tnere would be little negative reaction to 
over-structuring and tnat the data collection process should be almost 
totally predetermined and should result in standard reports tailored for 
fornative evaluation. 



Software Cot\$i derations' 



Tnti two asnects of the AIS software which most stronqlv influenced 
duthorinq systen desiqn decisions were availability of the CAJIIL 
pronrci^r.im Unnuaqe and nast experience with the use of interactive 
data base editors. 

The v^A-llL Lanquaqe. CAMR (Computer Assisted/Manaqed Instructional 
Unquaqe) {Pflasterer, 1973) is a higher level, 'jeneral purpose pro- 
nranmini lanquaqe developed as an inteoral part of the AIS» Several 
hundred CAMIL proqrams are currently operational and are used for all 
on-nne phases of the system. 

In specify inq a lanquaqe for AIS» it was determined that it would 
need to be Dotn CAI and CHI oriented and usable by personnel with a 
wide ranqe of experience. If support requirements had been limited to 
LAI. a lanquane such as TUTOR (Sherwood, 1974) as implemented at the 
University of Illinois, could nave been considered while traditional 
lanyuaqes such as PL/I, FORTRAN and COBOL coulc* have been considered if 
CMI nad been tne only tarqet application. Since both applications were 
to be supported and no known lanquaqe contained specific capabilities 
for both, a new lanquaqe was considered necessary. 

To accomplisn its diverse goals. CAMIL is divided into two sub- 
lanquaqes: Basic CAMIL and Extended CAMIL. Basic CAMIL is oriented 
pri'narily toward systems and applications proqranmfnq tasks. It pro- 
vides a full ranqe of standard data types includinq INTEGER, NUMBER, 
L'.GICAL. CHARACTER. STRING. TEXT, CLASS and SET. The data structuring 
nietnods include ARRAY and RECORD types. The flavor of the procedural 
conponent is similar to otner ALGOL-based lanquaqes: procedures are 
easily defined; the scope of variables is controlled through the use of 
blocks; and statements can be combined into logically meaningful groups 
through tne <-ie of compound statements. 

Extended CAMIL is oriented toward the rapid development of large 
proqrams. Its nost important feature is the sentence facility wnich 
allows coding in natural language-like statements. Also, fewer pro- 
granmer connents are required because of the level of docurnentation 
Which the sentence facility provides. By using sentence declaration 
statements, new sentences can be created by defining new words— nouns, 
verbs, adverbs, adjectives, and prepositions. These words can be 
combined in any meaningful way to form a large set of sentences. Hew 
data types and operators can also be defined. Sentences, extended data 
types, and extended operators can then be placed iit a CAMIL library. 
Extended CAMIL Programs can also utilize Basic CAMIL, allowing the user 
as much control as experience permits. 

A total system approach was adopted to make the language reliable, 
efficient, and easy to use. This system consists of the language, a 
compiler, loader. Interpreter, souixe program editor, data base file 
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manager, and proyram archlver. The combination of la/iguage. compiler, 
and source program editor facilitates efficient creation and mainten- 
ance of reliable CAMIL programs and reliability is enforced during 
execution by the interpreter. Programs interface easily with the file 
manager to rapidly perform the extensive data base management required 
for CMI apuli cations. 

Due to the nature of the CAMIL system, large amounts of error-free 
code can be written and checked out in a relatively short time. Given 
the structured nature of the languaqe, the enforced modularity, and the 
. interactive environment, relatively few logic errors are encountered, 
unce a progran nas been compiled, it is almost certain to run properly. 
Finally^ he Extended CAMIL sentence facility provides a powerful pro- 
gramming and documentation tool for relatively inexperienced 
proqrar.iiiers . 

AIS experie nce witn Interactive. Editors . While the AIS CMI function 
are supportea oy CAHIL software, the day-to-day operation of the CMI 
system is controlled through a set of data base editors. The intent of 
tnese editors was to allow course personnel with no prograwning 
language skills to define the characteristics of their courses and 
establish tne rules by which student assignments are made. 

The approach taken was that of a totally table-driven system. At 
eacn point where an Instructional decision is required, an attempt was 
■^jade to allow for a variety of decision rules. The indication of which 
rule was to be used (and in some cases, the rule itself) was then 
treated as a G.ita iten rather than beinq coded in the software. Thu?. 
botn tne flocision rule and the information to be processed by the rule 
Uy:*icdll/ a conoi nation of student performance data and course 
characteristic Parameters) are considered to be data. With tnis 
api^roacn. software cnanges are only required wnen the basic operating 
philosou'v of tne system is altered. .Nomdl' operational changes in 
course content and configuration, resource inventories, and student 
assiannent selection rules are made by changing the course data base via 
tne interactive data base editors. 

Jperational experience with tnis approach nas been quite positive. 
Relatively few software changes nave been necessary to meet the 
systeri's evolving instructional requirements. Uespite the complexity 
of tne data base, course personnel have been able to use the editors 
to institute data base changes appropriate to their needs. This is 
not to say. however, that the use of the editors is totally simple and 
straiohtforward. The data base and the interactions among its com- 
ponents are complex. In some cases, this complexity was not adequately 
taken into consideration in the human engineering of the editors. While 
all of the editors prompt the user's input to some decree, extensive 
pronpting was sometimes sacrificed in favor of effi ^ency. In 
retros:>ect. this trade-off was often inappropriate. 
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In nenerdl* nowever, the interactive editor approach was thounht to 
holU considerdole Dronise for facilitdtinq at least some aspects of the 
v-kI aevelopnent process. It was also recognized that where such an 
am roach was •iidcpted* it would be desirable to provide extensive 
proiiU>tuv;. 

Prior Kpproacnes to tne Autnorincj Problem 

M jrowinQ recognition of the problems associated with CAl develop- 
tent has resulted in a substantial literature addressing these problems 
and proposit.q alternative solutions. The following sections provide a 
brief irttrotiuction to the concept of authoring systens, an overview of 
tne ty:>e!i of existinq systems* and a sunwnary of two Air Force 
experiences in CAI development. 

Autnorin-j System Considerations and Criteria . Zinn (1974) lists 
four criteria for assess inn tne effectiveness and utility of CAI autnor- 
Jrt} lanqudoes: reliability, efficiency, flexibility, and convenience. 
Tnese sane criteria can be applied to the more general concept of 
iutnarin.: syste^s. 

Jr.ier reliability, litm includes automatic recovery for both autnor 
aiij itucer.t foHowinc) systen failure, linitlnq tne loss of authored 
•wterijl dnu 1 i"".! tinq the domain of author errors, i.e., an error In 
one part of a program should not Impact other parts of tne program or 
'jt^ar pro}rams» Efficiency refers to both tne time required for 
aufiorirn a»>'i f^d co"iputer time required to translate author lanquaoe 
stater\er.ts into executatlle code. Flexibility considerations include 
access to a variety of devices, access to alternate modes of execution 
or convertinrjs and the capability of adding new operators, stateiiients» 
anc subroutines. 

Mutnor coT^venience is treated as a major consideration. 21 nn 
sug«)ests tnat the language (or system) should have a minimum of 
retlunduocy and irrelevant syntax, e.g., the program listing should be no 
•iore complex tnan the author's actual task. There snould be provision 
for alternative authorinq styles, e.g., while many authors indicate a 
preference for interactive entry, others prefer to work with paper forms 
wttich remind tne author of system capabilities and requirements. With 
respect to revising an existing program, Zinn notes the advantages of 
on-line editing and suggests that the system should provide access to 
the original file, use straightfonwrd notation for determining changes 
to be fnade, and confirm that changes were accomplished. In testing a 
program, the author should be able to begin execution at any point and 
trace througn the program using labels as indicators of location. The 
language notation should help a reviewer understand the intent of the 
instructional content and strategy. Finally, access to system 
capabilities should Increase with experience. 



K.a p ow (1975) states that in order to maximize assistance to the 



author, u is not sufficient to simply add authoring aids to a pro- 
grawning language. Rather, the total CAI system must be organized 
around thu goal. He then describes what he considers to be the basic 
features of such a system. 

First, the system should provide a structured format to help 
authors organize their concepts. The structural units should match the 
author s conceptual units and Impose a degree of modularity. The 
current working unit should always be identified and the authors state- 
ments shou d refer only to this unit. Kaplow suggests that the 
operational aspects of the authoring and CAI delivery components should 
oe separated. A CAI program should be treated as a data base containing 
the content and structure of the program as well as the computer 
comands to be executed. The system design snould make it explicit that 
the program is a collection of information organized so as to be 
amenable to understanding. The computer itself should automatically 
perform many progranminq functions, such as checking structural complete- 
ness and findinq cross-reference errors. Finally, the system should not 
require tnat a program be complete before it can be tried out. 

In nis subsequent discussion, Kaplow makes a number of additional 
critical Doints. The system should be tolerant of user errors and pin- 
point errors at tne time tney are made. The detailed actions to be 
taken at student run time snould be defined on the basis of the 
implications of tne autnor's instructions rather than having to be 
spelled out. Ciiven a modular program structure, the system should nelp 
the author keep tracK of the interrelationships between the various 
parts. Tne fact that it is often easier to write a new program rather 
man to t!K)Uify an existing one is particularly unfortunate considering 
tne opportunity, even requirement, for CAI revisions based on student 
performance. Since the source of this problem is usually one pro- 
qramer's difficulty in understanding the structure and logic used bv 
anotner. tne systen snould place particular emphasis on the ease with 
wtncn one can build on existing material. 

Lxample Mutnorinq Systems . Tne earliest CAI programs were written 
tne available qeneral purpose lannuages. Altnough this Is still a 
popular approacn {e.g., the extensive use of BASIC), there was early 
recognition tnat authoring cou' be facilitated by languages tailored 
to the particular requirements of CAI. Consequently, there was a pro- 
liferation of CAI authoring languages— at least. 30 by 1^73. The last 
decade has also seen experimentation with various author entry systems- 
approaches wnich are relatively independent of a specific language. Tne 
following paraorapns briefly describe an example of a modern CAT author- 
inn lanrjuaqe (T'jTuR) and sense approaches to developinn authoring systems. 

TUTOR (Snerwood, 1974) needs to be considered in the context of the 
complete PLATO { Proqrafrwi ng Lonic for Traininq Operations) system. The 
systen provides for interactive entry, easy trial and revision of code. 



dtu IS juue resDonsive to authors* needs in terms of dlsp^a^ time, 
coroiUtion tiTJo. and diannostlcs. The author's task Is facilitated 
t*y a nuntcr of aids, such as on-line access to reference naterials, 
san{>le »»ro^^rdn routines, and files of current Uocunentation and coments. 
To a larue extent, the PLATO ap.iroach to authoring is based on the nodel 
of dn dutnor woo is a versatile professor, an expert in the subject 
matter, an ox.>ericncod teacher with sound but innovative ideas about in- 
structional ►iresentdtion, and a capable pronra»nner. Uithin the uni- 
versity environnent. tnis apuroach has resulted in a larqe number of 
tixcelK-ni lessons. In otner environments, where the autnors have 
been less experienced, less Skilled, and/or less motivated, the approach 
hds not proven as satisfactory. 

>'ovjSey (1 j7h) describes five cateoories of approaches to builUinn 
easy dutnor-entry systems: separation of loqic and content, avoidance 
•y any dutnorirv: lin'iaane, use of lesson plannimj guides, conversational 
sattridU :enerat1an, and nacro systens. The approaches in each 
v-dtL-.'^ry :etu to uuild on the concepts of the prior cateciories. 

"hiost all of these approaches er.rploy the tactic of separatinn in- 
strjctiondl content fron proqram loqic. Tnis divides the authoring task 
in a way that is particularly amenable to a team approach. Dowsey notes 
mat to oc effective, such separation requires similarity of structure 
between t^e two components. 

The Mo-duthor-1anquaqe approach not only separates content and loqic 
t'Ut does not require the author to define the loqic. Only the content 
Is speci^'ieo. in the form of franes or problem cateqories. Tnis is then 
acted upon by a lesson generation proqram which treats the content as 
Odta to produce Instructional materials. 

Tae use of lesson planning guides, while requiring the services of 
a coder, pernits tne author to conmunicate with the computer in English, 
Typically, tne author defines the material to be presented, questions, 
expected answers, and corresponding courses of action on a standard form 
wnicn specifies tne cateqories of information required. One of the more 
sopnisticateu examples of this approach is Dowsey *s own COURSEt-lAKER, 
designed for use with the COURSEWRITER III language. It is based on use 
of a paper form which includes a presentation section containing the 
fvdterial to be displayed, a question section indicating the student 
response (s) expected, a decision section defining whether a branch is to 
be taken (and if so, the type of branch), and an analysis section con- 
taininq response judging rules. 

conversational materials generation represents a quite different 
approach. The interactive system assists the author by eliciting the 
content and logical structure of the lesson through a natural language 
conversation. Paloian (1974) describes one example of this approach. 
The author defines the program structure by' entering a sequence of action 
verbs, Tne system checks the accuracy of the sequence and, if it Is 



2-1 



correct* prompts the author to supply the text, anticipated responses 
counter names, etc. 



uowsey sees the use of macro routines as potentially beinq the most 
powerful approach. In usinq such a system, the author retrieves a pre- 
proqranitietl sequence of code and specifies arguments which complete the 
routine^ One characteristic of this approach is that It Imposes a 
definite structure on the material produced. The Time-shared, Inter- 
active, Computer-Controlled, Information Television (TICCIT) system 
(Stetton, Volk Bunderson, 1973) is probably the best known example of 
a macro systen. TICCIT employs a single Instructional strateqy, oriented 
towanf concept learn Inn. which assumes learner control over sequence. 
The strateqy specifies how tne subject matter should be structured (e»g., 
rules, examples, practice items) and even suggests the appropriate 
number of items in each catenory. Thus, much of the author's task 
consists of iwldinq tne content to fit the strategy. Planning guides 
are useu to fomat the content and define presentation tactics by 
selectim ^nonq available options, A macro processor then converts this 
tnfornation (content, content format, and strateqy options) Into computer 
lanuuane. 



une other nacro-oriented aoproach wnich deserves mention is the 
use of ::unoforns {^jchulz. 1975) developed for use by military authors 
on f^L^T . U". Tnt- 'lonoforto macros, written in TUTOR, were intended to 
facilitate .^roDaration of frequently used question types by providing 
d question fomat and el i».nnatinq the need for the author to understand 
TUTJU. A lotal of nine ''wcros were tievelotwjd for malt in"Je choice, 
construLteq resronse, and mdtchinq questions. Viitnin each question type, 
th(> "lonoforris di^^er with resr^ect to variations in format, tyi^e of 
ffedhack and (for tiultiple choice questions), order of alternative 
nrpsentation. Tnere are also a number of ontlons within each Monoform. 
The autnor copies the desired Monoform and follows the Instructions 
(S'jnnlied in tne form of program comments) to su».'ily the question content 
and tailor the TLTvJH conr.iands to his/her snedflc requirements. Schulz 
rerorts tMdt use of .'lono forms reduced the 2 to 6 hour question develop- 
ment ti'ric to onl/ 10 to 15 minutes. 

Air Torco lAI Authoring Fxperionce . Himvnch (1977) reports a 
comparison of TICCIT and PLAto authorinq efficiency conducted at^'taxv/ell 
A^i.. Tho results of the comparison were inconclusive vnth little 
difference found i.t.'twt.'en the tm approaches. In the production of 32 
stjuent contact nours of CAI, the PLATO team required an averaqe of d22 
uannours ')er nour unile the TICCIT team required ^4o. Uf greater 
interest is ►iii.iwicn's description of tne procedures followed and 
problems encountered, particularly by the PLATO team. The training pro- 
vuitu for PLmTu team menbers consisted of an intensive i week session 
anu iuosequent continuous consul tinq support oy the PLATO staff. TICCIT 
tca.i training ncian wi tn 7 weeks of familiarization followed, some i-ionths 
later, by J wc»eKs of intensive training. In both coses, the traininq 
ro({uirrU appears excessive to tne autnors of tnis report, further. 
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ni'iw^cn reports tndt in many instances the authors did not capitalize 
on PLhTs's floxiMlity with resnoct to instructional strateqies anU, 
consequently* aio not denonstrate the systen*s full capabilities. 

uall'hin^ jeLeo, 'tain anc ulllnan (1977) provide a comprenensive 
description of an experiment in the use of KlWj IV for Air Force 
tecnnical trainin-u At the beqinninn of the test, conducted at Cnanute 
AF:>. the ty^ocil PLATu authorino nodel was adopted with each author 
actinn tnJependeiit 1y. Learninq Tt^TuR was found to occupy a niajor portion 
of authors* tine. Authors nau varyinq styles and quality control 
stanuarus* ana dS a result, the curriculum was fragraenteo with little 
continuity between lessons, Jallman et al. concluded that a basic flaw 
in this approach was the unrealistic assumption that all materials 
authors were exf>ert? in both Subject natter and instructional practices. 

A tean a^-siroac^ was subsequently adopted in which the team con- 
sistea of m autuor, subject matter expert, instructional proqranmer, 
auvj coaer. ^Uriouih tnis was a distinct improveraent over the prior 
dr»»rodcn» a nu-u.er of problems were still encountered. i4o written 
:>rocedures were defined, only infornal understandinqs amonn tear nenbers. 
Trn"; res>ilted in ti"ie consupinq coordination uroblens and inefficient 
use of tear- specialists. Administrative and manaqement procedures were 
ri*jver well defined and there was a continuous need, never resolved, for 
better, rnore extensive author traininq. 

A number o^ PLATD features wer. found to be quite useful. On-line 
duta collection routines supplied by the PLATO staff and the capability 
for on-line text editing were both considered important. TUTOR was 
aderjdte ^or tne site's needs with authors handlinq the simpler aspects 
of proqrdMMinq and coders required for only the more conplex portions, 
^nly a few of tne more experienced authors, however, capitalized on 
PLATO'S instructional flexibility. Almost all of the lessons produced 
employed tne sane simple tutorial model. The report authors suggest 
that this ai-proacn was followed because the materials were easy to 
prepare, tne subject matter was not that complex, and student com- 
prehension and. retention requirements were low. Branching was used 
;;>ainly for forced review and TUTOR'S response judging capability was 
seldom utilized, only about 20 percent of the questions developed 
ef.tployed a constructed response format. Not only were constructed re- 
sponse questions more time consuming to code, students disliked them 
because of unfamiliari ty with the typewriter keyboard, 

Jallnan et al. drew a nvwJber of conclusions relevant to the current 
project. The team process was found to be more efficient and effective 
tnan individual authors. Authors did not exploit PLATO's full 
capabilities due to resource constraints, lack of CAI expertise, and 
inadequate training In instructional progranriing techniques. Finally, 
sophisticated CAl capabilities may not be necessary for effective learn- 
inq of the type of tasks and level of knowledge required for adequate 
''Military technical training. 



The types of problems encountered at Maxwell and Chanute appear 
typical for military technical training* Kimberlin (1977), describing 
the status of Project ABACUS at Ft. Gordon, reports ii to 6 month slips 
in the full implementation of CAI courses. The major problems were re- 
ported to be changes in the Plans of Instruction during CAI development 
and the fact that t'ne project was never assigned an adequate number of 
instructional progranmers. 

Uesign Conclusions 

Although many aspects of the military technical training environ- 
»nent are not amenaole to efficient development of instructional 
materiaiSj particularly CAI, it was concluded that it was preferable, 
at this time, to adapt the system to the environment rather than to 
develop an authorina system which assumed a mure responsive environment. 
It is hoDed that the authoring tools would themselves act as change 
agents. For exannUs it was decided that the authoring system should be 
desiqned so as to promote the concept of an authorinn team but not to 
rely on its existence. Given a trade-off between authoring flexibility 
and a structured ;»resentation format, structure was almost always 
selected. Jifficulties in assuring adequate author training was accepted 
as d major problem. Despite the fact that many features of'CAMIL were 
designed expressly for this purpose, it was recognized that any approach 
wnicn relied on an author/prograrnner had little chance of success. 
Furtner, U was considered unlikely that personnel would be assigned to 
such d sj*ecial izec function as CAI computer programming in sufficient 
nu.;il>ors to adequately support many authors. 

Inis assess"ient suggested either a no-autnor-language approacn or 
a Jidcro s/ster.i approach. The former was considered to be toe *1qid for 
tne evolving comjuter-based tecr.nical training environment. A macro 
syster; nooeled on tne TICCIT approach was rejected because a simpler 
aucnorin-j process was aesired; tne sane objection was encountered for 
dpDroacnes si-nlar to itonofoms. Tne adopted design combined the best 
aspects of several prior approaches, relied heavily on characteristics 
of t?»t existifrj uAllIL system, and cdDitalized on prior AIS experience 
witn ii.toractive data base editors. As suggested by Kaplow (197b), the 
total systeri was Uesinned with the major goal of facilitating CAI 
naterials development. 

To alleviate the need for computer prograimnng, tne aPP»*oach 
aaopted wds to arovide a small number of flexible "template" CAI pre- 
sentation prmrans, each of wnicn would support a class of CAI wdules, 
While this amroach still requires the services of a computer programer 
ti exMtul, tne extent of thes<* services v/as reduced in two ways. First, 
V t: ttM , late 1 rograns wtjre Uesiqncd to be as flexible as possible with* 
o;.t D».ini unduly conplex. Princiules which nad been employed in 
it'vt loi.inq the hIS Ad»\ptive Model and its data base were used to assure 
tridt J sinfjle pronrap structure would support a variety of modules 
auuressin'j different topics. Second, given the characteristics of tne 



CA.*UL lanquaqe and a template program which was well structured and 
documented* it was anticinated that a relatively inexperienced programmer 
could modify the basic nronram to meet the requirements of new 
appl ications. 

Tne major emphasis was placed on designing an interactive authoring 
editor by means of which an author could define CAI content and branching 
lotiic. The desion qoals were to provide authors with a variety of 
options within a structured framework, to appropriately guiue and prowpt 
authors uecisions with respect to these options so as to minimize the 
neeu for formal training, to automatically accomplish as mucn of the 
prograiuinrj detail a^ feasible, and to eliminate any need for authors to 
oe aware of tne computer language. The forrnat of the authors' input 
was to be as similar as possible to what students would actually see, 
Autnor errors were to be detected and identified at the time they were 
coMTiitted, Standard routines and reports were to be provided for student 
performance data collection and analysis. Finally, the system was to 
provide convenient access to meaningful information wnich would assist 
in nianayinq tAI development. 
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in APPROACH 



technirti ?rr?ML supporting efficient CAI production in the 

AuthoHnn lul'l ^ environment centered around development of a CAI 
roan^rial acaur.??tn Presentation Program. Student perfor- 

a^re^rts S^vltliln r?"rr ^'^^ 5^'"'^^ presentation program 

A CAI^r^riffri^?!? S ^'^ °" ^^'^^tive evaluation requirements, 

inas n?^?!r^L^i ?^ S^^^""*"^ ""^^ developed to provide hard copy list- 
InnL^^^ ^ nwtenals for author and student use. All b"ut one of the 

?rSg?an wlrrSL'^^^^^ ^'^"f ^" ^^^^^^ ^^IseJtation 

1 ^ pperate on the current AIS interactive 

temmals. wmch are a modification of the PLATO terminal! but provisions 

T^TtaTttZ^'l'- ^T'^'^"" ^^^^^'^'^ teminats rL^Sring 

tne ^la^.riafr^^rrD^'^^^'^'''- Acquisition and Analysis Progra..is. ^ 
tne .latertals Print Program, and documentation of the authoring system 
are uescribeci in detail in the following , sect ions. ^"^^^''^"9 ^^^^^"^ 

i< ../r^'!!:^^^-^' tne software supporting tne AIS CAI authoring syste;n 
IS an interactive Authoring Editor. To introduce the Editor, tne 
structure ana characteristics of the CAI modules produced with it are 
tirst aescnueu. Tne Editor itself is then discussed, beginning with 
various sur.r.ary displays wnich provide central reference points for the 
autnor 3 work. ,ne mechanics of generating text and question frames 
some ac-dir^""^ t>ranching logic are then described and illustrated in 

, ''^^^■-'^e Structure. Fra me Types and Frame Flags . As structured by 

-^tnoriny cuitor. a CAI module is segmented into objectives. Through 
une Autnoring Lditor, the author can define only 'the sequence in which 
objectives are to be presented. Any branching between objectives is a ' 
function 0' the Presentation Program. Eacn module must include an 
Objective J tnat contains material (usually an overview) pertaining to 
tne lesson as a wnole. The author may then define up to 100 numbered 
objectives containing the module's instructional content. Each objective 
i-an contain uu to Uo frames. Ten frame types have been defined which 
can De classified into three categories: textual content frames; 
question frai.ies; and special purpose frames. New types can be defined 
as requirements arise. The frame types in each category are described 
briefly below. - 

There are six different types of textual content frames. These are 
an siirtllar m tnat they simply present textual information to the 
•^tuuent and require no response other than an indication that the student 

. ready to proceed. The six types are differentiated primarily for 
»ianaqewent reasons. In some cases, a specific frame -type may be re- 
quired at a certain point in the frame sequence and listing frames by 
type provides a quick oVerview of tne instructional sequence; 



Tne udSK "Ttixt" frai.ie is» as its name implies, the pritnary medium 
for prt'sentin^; instructional content. *a Text fnine can contain up to 
four "paqes" or screen aispUys* There is no provision for branching 
wUnin frames, i.e.» between pages* unce a frame has been selected, all 
of tno ,)a'ies In tnat frame are presented. Authors are instructed that 
d Text fraffie s»ioulci >jresent a discrete chunk of infonsjation pertaininq 
to a sinqle concept. 

Tne second textual frame tyoe is the Elaboration frame. It differs 
"frpia a Ttixt frame only ty name and intended funct1on«-to present Infof- 
:?iatioLn to students wh^se perfcrrmance indicates that they need further, 
more dK^iled explanation of a concept presented in a Text frame. The 
Title frame>,wnen used for the purpose implied by its name, can be 
recognized ds^a^ividinn point in the frame sequence. The Statement of 
Objective f ra'ie rs- xrequi red at the start of each objective other than 
Jttjectiy*: V. Sir.ila>ly» an Overviev/ frame is mandatory within Objective 
J. Firtally, a '^'ateriat's List .rame, llstim] any reference materials 
wnica tn.' stuuent needs to conD,lete the nodule, is required in Objective 
J. Any of tne nancatory fra'me types can also be used, at the author's 
ui5>cretion, at otner points in tstie module. ' 

The Eaitor currently supports twt)N^pes of Question frames: 
'ultu;le cnoice and constructed response>\ These frame ty^es provide the 
:>r1;-iary icarss of evaluating students' . under^tandinq of the material being 
prosentou. .'jestion frames consist of the question sten, up to five 
.';niUi.'1t'-cnoice alternatives or nine anticipated constructed response 
aUtrrnaiives, feedback statements for each alternative and oronpt state- 
:!»ents associated with successive attempts to answer the question. 

Tntre are two special purpose frarKj types, neither of which is pre- 
sonteu to stuuents. The first is Uocur mentation frame required at the 
begihninq of each objective to provide a means of documenting the 
authoriruj/evaluation/revision process. The author is Instructed to pro- 
vide infor.idtiofi such as the dates during whicn the module was being 
developed or revised, the names of additional authors who have worked on 
tne inoaule, tne learning strategies employed, the nature of and reason 
for any revisions, etc. It is anticipated that the information requested 
•will become more structured as a function of experience. The second 
special purpose frame type Is the "Branching Decision" frame. It is a 
dunny frame, containing no material, which allows for a branching point 
without anything being presented on the screen. 

Tnere are eight varieties of "flags" which can be set against a 
frame to indicate to the CAI Presentation Program that a particular 
action i'S to be taken when that frame Is encountered. Like the frame 
types, flags have one to two character code names which, when displayed 
of» the frame listing, irontrlbute to providing a sunmary of the irutiruc- 
tloital strategy employed. 

"Four of the flags (Objective Passed, Objective Failed, Lesson 
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passed, ana Lesson Failed) notify the Presentation Program that. If this 
frame is encountered due to author«defined branchinn, the student has or 
nas not coopleted the objective or lesson satisfactorily. The SMp This 
frame flan indicates that the frame, even when encountered, is not to be 
'Presenteu to students. This' allows the author to store alternative 
material in the body of the rnodule itself. The Fix Answer List flan, 
which can only be set aqainst a nuUiple-cho'lte question frame, indi- 
cates ..nat tne list of alternatives is to be presented in the order input 
by the author rather than being randomized. A Student Break flag elicits 
a display generated by the presentation program sunqesting that the 
student tdKe a snort break from working on the nodule. It allows the 
^athor to alleviate student fatigue. on long, complex modules. Finally, 
the Decision Point flag indicates that the presentation program is to 
collect uecision Point student perforr.iance data following presentation 
of tnat frame. 

. . '■Q<^^le, JD jective and Frame Sunn^ary Uisplays . Throughout the 
Lditor. a ;miiosorhy was adopted of providing extensive prompting and 
.ail-safe" iecnanis.TS. Lach author entry is prompted to the extent 
possible. v,o.rianjs wnich will result in tno deletion of materials or 
Drdncnirin iDiic require a second "Yes, I want to do it," response on 
th,.. ;^drt of the autnnr and it is always possible to back out of an error 
situation without the Editor takinq any action. 

Wnether an author wisnes to create a new CAI module, revise an 
e)^istin--; -lodule or disr.lay a nodule, the sane basic procedures apply, 
'tter accessin'r tne Aythorina Lditor fron an interactivt terninal, the 
aut^or L-fiCQui-.ters i display, illustrated in Fioure 1, requestinq tiie 
so.;u?r.ce nu-.:;rrrs (Course '.unU.'r throuqn Module Nu-nber) identifying tne 
particular loUjle to be accessed or created. Entry of each value is 
►•ror;;tod ani fie ran^e of allowable values is specified. Tne author may 
also rt.'uuest a list of existing nodules (by nressinn Function Key Fl). 
I. this oi'tion is selected, tne author is ^SKed whether all niodules or 
only tnusc Delominq to a particular authw. are to be listed e id wnetner 
the list IS to be displayed on the terr.inal or printed on the central 
lino I'rinter. 

I^. tne iuentifier of an existing module is entered, tne author is 
dSKed wnether tnat nodule is to do displayed, ciianged (revised or adued 
to), or deleted, wnereas any nodule can be displayed, only those created 
oy tnat autnor can be changed or deleted.. If the author enters the 
iuentifier of a module wnich does not exist, tne author is asked wnetner 
d new nodjU* is to be created, and if so, wnether the module Is to be 
a copy of an existing ::iodule. Tnis copy feature allows an autnor to 
revise a rx>dule which nas already been impleiiented in the classroom, to 
revise another author's nodule, etc. If the new module Is to be created 
fron scratch, the author is required to enter i^ha title of the module 
and tne nunbcr of objectives wnich it is to contain. 

' If tm> autnor is creatinn a new nodule or revising an existing 
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module* the next display asks which data collection routines are to be 
activated: response data, decision point data, and/or student eonroents. 
None, any, or all of the three types can bfe selected. 

• The author next encounters the Objective List display illustrated 

in Hqure ^. Tnis display provides a central reference point for the 
author— a top level overview of the module's- content and the control 
point for creatinq, changing, or listing complete objectives. Objectives 
against wnicn no material has been entered are shown as being undefined 
(Undef). For the display illustrated in Figure 2, the module includes 
the jnandatory Objective 0 and four fidditional objectives, two of which 
are as yet undefined. ; 

Actions wnicn can be taken at this point are listed as Options at 
tno bottom of tne display. If the author enters the number of an objec- 
tive, tne list of tne frames defined for that objective Is displayed. 
One can also aca or delete an objective, reorder the sequence in v^hich 
the oDjeccives are to be presented, or request a printer listing of one 
or nore obj« Mves. As is the case for all of the Authoring Editor dis- 
plays, tne Qjthor can back out without takinq any action by pressing the 
"BACK" key. 

If d new riodule is being created, all of the objectives would be 
sr>own as undefined, and the author would first be required to define 
Oocunentation, Cverview, and Materials List frames for Objective 0. The 
author cQulu tnen define additional Objective 0 frames or return to the 
u.jtctive List Uisnlay and proceed to define content and branching with- 
in tr,«.' uuhvidual, numbered objectives. 

rinen itie a-itrior accesses a specific objective, the first uisplay 
encountered is the Frame List illustrated In Finure 3. The Frame List 
prov1«it?s d central Doint for work within an objective. The list defines 
tno sequence anu types of frames coinprisinq the objective and provides 
an overview of the branching logic and any special conditions imposed on 
si/ccific franes. Actions which can be taken are listed as Options at 
tne tiL»ttoi: of tne display. 

A Frar.ie List consists of one to four j»ages with up to JO frames 
listea \ivr piqe. Tne objective represented by the Fratiie List si»own in 
Fi'jurt.' 3 is relatively short, only fra»aes. Tne display provides a 
substantial amount of infornation about trie con,teiit and instructional 
strateny e!:ployud in this objective, Lach- frame nas both a number (1 
tnruuun c.,) ami a name consisting of one of two letters (e.g., D for 
uocu'ientation, S for Statement of Objective) and one or more digits 
indicatlfiq wnethor this was the first, second, etc* , frame of this type 
to be defineu. I* there is branching logic associated with a frawe, 
tnf class of Ionic (Pre-frarie, After-frai-^,* or Response contingent) is 
Indicated, anu limits on the number of attei-'pts allov/ed to ansv/er 
questions are shown, as are Frame Fla^is. - 
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Figure 2. Authoring Editor Objective List Display. 
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Figure 3, Authoring Editor Frame List Display Showing 
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>e oajfctive lUustrateti \n Fl'^ure j begins wUiMho fe;;uireu 
L'rtitior ->r^e (Jl}. a StaU-ietit of vH.juctlv^ (SI), a Title (T^U 
41.4 tin iverview fraiie FoHowInq a simile. Text franie (T1), tne 

dutrmj* tests tne student on a .-JrereMulsUe concept* usliifj two laultlpTe 
i.»«<nctj ,jt>t:ionb p"l jiriu .«»ly two dttOMpts are allowed to ai^swer 

vMcn jucstion. Tue stconJ Muestion Is followed uy { After- fraiie) branch- 
nr. Ivi-ric rt?nc'» routes the student around a review o7 the conce,it (franes 
t'lrokijn lj)-if .yrfonitince on tnt* tv^o questions Is auequate. Only tne 
oxts tehee -of tne loqic Is sncwn. not its detail. The review consists of 
thret* fe)ct fra«ies and two fiore .Multiple choice Questions. Uote tnat the 
ni>to -jf Yrarse nu^b-.-r 10 Is TU\ This Indicates that the frawe was added 
lat.;r,laftcr fraMe Til had been defined. The frane 1^ After-frane loqic 
ueter' jnes whether the Student nas now mastered the concept. If not, the 
stuuyntl IS retarr-iJ to frane nunber and to review tne concept a second 
tMt:. ,.n^ifs *^rane n-inber 11 Is encountered the second time, tne Pre- frane 
lo-ilc i w Mssvs tne^aultnile choice questions and routes the student to 
ra- >• -sIj; r U. To aeteniino now many students are beinq routed tiirounn 
l-'.o'jr r^vkv. fra'n*»s and th" tine required for review, the author nas de- 
fir.t:v: ..-icislon P.jint rians {j) prior to (frame nunber 7) and followlno 
'vtra-v liu-ibvr 12} tne review sequetjce. 

Tn(4 -lain bocy of Instructional content Is presenteo by frame nu;)- 
>>ers U I'vro^nu c*.. cunsistinu of Text, Llaboratlon (E), and constructed 
r.."j,otiSt ^.L; frdiifcs. Since no Specific Unit has been placed on the 
»iu.a>','r af atteru.ts for answerln-; cnese questions, the default value of 
ir.roe jtie:'; ts will bo allowed. Response Lontinq en t branchinn Icjlc has 
jt.t:f. uk'tif\i.ii f.jr frd!.e ')CK If tne response batches a particular antl- 
cv. it ir.C'jrruc: respoi>st;, tne stuuent will be routed to Llaboratlon 
-rj >«^ .0^1 ;:..„»i to correct tne nisconceptlon . Othervnse, fraMe LI Is 
•-.».!,.,>. : jin t i^' .-ll •\fter- frd-..e brdnch, 

!u.,tk rv 0* the objective Is evaluated by six snultlple choice 
luest>of>s J le numbers througn iJO), Unly one attempt Is alloweu on 
edc% ,j^st' . Tne criterion for passin'j the objective Is four out of 
six currect, and »• tne author nas defineo After-frame branchinq logic 
t"n] lowUi'i )»\, to SKir) tne last two questions If the first four were all 
correct, -.merwise, the student's uerformance Is evaluated foUowinq 
frj; c n If the criterion was not met, frane T9 is presented to in- 
forn tne Student of this fact. The Frane Flag (OF) indicates the 
occurrence of in objective failure to the presentation pronran. If the 
criterion was net, after either four or stx ques.tions, frame TlU Is 
presented v/hlch congratulates the student and tne OP Flaq notifies the 
Presentation Proqran that the objective has been passed. 

The actions which the author can takSa are listed as Options at the 
nottot"! of the display (see Figure 3). tntry of a specific frame nunber 
accesses that frane for the authop's ♦•i.view or revision. Option I (In- 
sert) allows the author to define a new frar.ie and insert it at atiy point 
In the frame sequence. If the Insert Option is selected, the author is 
dSKed to prdvide the number of the frame after which the new frame is to 
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be inserted and to select the Type of the new frame. The Frane Type 
selection nenu js shown In Flqure 4. Once Frame Type nas been selected, 
the author IS .Ucej in the frane creation node for that type? Note 
that UMUon 11 on tne Fr^ne Types -lenu allows tha author to cipy ar" 
exjstinq frane. my frane, includim one frou another author's proqran 
can bt. co.ne.l. Furthor, the autnor has the choice of makinq an actual 
copy of tne *r.in»e, wmch can then be nodified to suit tne author's 
^ purposes . or of sitiply "aliasing" an existinq frafne. With the latter 
option, tne frdt,«i IS not actually copied but rather, wnen an "alias" 
frane is encountered by the Presentation Proqran, the frame content is 
retneveu fro-n us orlninal location for display. 

.uthJ''^/'c'^^'"'^'^^•^''^?^ !:?^^ ^^^""'^^ ^^^^^ (H. allows tne 

autncr to vet tno previously discussed Frane Flags against specific 
fraPHis. Specify urancnint? Loqic (B). perriits the autnor to define 
brancinno loaic for suecific frames. Ootion C, Coment File, accesses 
the list of student connents which have been made aqalnst specific 

J^'^^'^^-'stence of such connents is indicated by the occurrence 
rnLill f ^^f'S column of the Frame List display. The Frame List 
•^ay consist of up to four nanes, and the control characters and 
acci'ss dis-lav of tne next or previous pane of the List, respectively, 
.ne je.ete Fra';e Option allows deletion of one or more frames which 

t^'n^J. 4^^r<:'!'"'.-^* '""''•^^^ ^'^^ ^f^^^^" P^»^"i^5 tne author 

to nove .ranes tn different positions in the *rane sequence. If the 

autnor decicies to ry)1dce all of a franc's content, the Replace Frane 
n.tion IS .w.nUE.le. Finally, the autnor can always return to the 
'u>.'Ctne List Jlsulay by press inq the BACK key. 

Text ::>enerat/on. The process by w'lich all six types of textual 
^.ontent fra:^es jiYe produced and edited is essentially the same. When 

autnor oU'cy. tn create one of th«sp frarue types via the Frame Tynes 
-onu v-iue n ;aref 4i, a blank tetaplate. sucn as Is Illustrated in Fioure 
^i^^iii'\/vJ, ^ Note tnat the frame nunbtfr, frame narie. and number of 
tnis t.a je wity.;- tne franc are shown at tne top of the dlsnlav as Is 
tne ract tnat/the author is in Insert rode and nas already entered one 
lino. Ty;i^j /content appears on the line next to the arrow-snaped cursor. 
M tat »i.AyKey is ,>ressD»J to end -.^ach Una. The author Is free to 
enter tr.v oSntent m any fomat within the constraints of the narnlns, 
maic^teci U/ tnu cursor on the left and the vertical line on the flqht. 
wucn i.a ;e /can contain up to 21 11n(?s of -jU characters eacn. 

Mft/r onterir-.f^ one or nore characters, the autnor can press iiACK 
I) acceA a colloction of editinn functions. This situation is lUus- 
tratej /n M jurc -o is the sa^'^e as thi» disi^lay wnlch would bo pre- 
senteyit t:'^«» ajfior accessed an existing Text frame to modify It. Tne 
•i.ntiy' .)ptir.ns .^r*? listed at the bottom of the d1s:»lay. 

The Insert Line{s) (I) Ootion allows the author to define additional ' 
tiy^s of text *nnow1nq or between ll'.^s o*" existinn text. While Insert- 
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FRftrt TYPES: 

1 Documentation 7 Constructed R^ponse Question 

2 St«t«ff>ent of Objective 0 Drenching Decision 

3 Overview 9 Title 

^ Tevt If risterisl List 

5 Elaboration u Copy of an existing fram^ 

6 Multiple Choice Question 

ENTER choice > 



Figure 4. Authoring Editor Frame List Display 
Showing Frame Type Selection Options. 
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nWC Si Ui PfiG£ t or I INSERT H006 i Linm 

» Thmm thm first coupt« o/ lif^T" T 

> typ«d by th« author . 



OP'TIONS: Enter *«ict material. 

Press NEXT to end each line. 
COPY keys copy previous line. 
Press OflCK vahen finished. 



iijure -J. Authoring Editor Text Frame Display as Seen When 

Creating a Frawe. 



nwrc $; 111 PPC£ I or i 

1 TH»« arc tht f\rst couple of 1 in«s 

tvped bv the author. 



^OPTIONS: I, insert 1 ine (s) r^lac<^ line 

0 ♦ de 1 et e 1 1 ne (s) B , draw/erase box (es) 
S» save line(s) F, fetch saved line(s) 

wipty save buffer 
♦ ^'•t disp I ay /change next /previous pag« of frafl« 
BftCK, return to frame selection 
fCXT, d I Bp lay /change text for next frame 
ENTER Choice > 



Figure b. Authorinq Editor Text Frame Display Showing 

Frame Lditing Options* 
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Ing material between lines, the two lines precedlnq and one line 
followinq tne entry are also displayed. 

The Replace Line (R) Option allows the author to edH one or more 
existinq lines» The oriqinal version of the line Is displayed, .the 
autnor s new version Is ihown below It (to the riqht of the cursor) and 
the preceding and following lines are also displayed to provide the con- 
text of tne change. A situation In which line 3 of a display is being 
eoucd IS Illustrated in Figure 7. A set of Edit keys facilitate the' 
euumn process by allowinn the author to erase, copy, or ternporarllv 
store tne conplete line, specific words or Individual characters. After 
the line nas oeen cnanged, the Editor redisplays the edited text and 
lOveS viowr, to tn^i next line to allow the author to- edit It. As in the 
Insert node, tne autnor can 3ACK out of Replace n»de at any time. 

The Jelcte l.ine(s) (U) Uption (see Finure d) allows the autnor to* 
erisv^ one or jore lines of text. The Editor prompts the author to in- 
dicate toe lines 10 be deleted (the numbers of the first and last lines) 
surroanos the indicated lines by a box for visual emphasis and asks if 
t'^o dutf.cr wisnes to complete or- cancel the deletion request. Any text 
^ri tno ^jaqe foUowinq the deletion is moved up to replace the deleted 
'att.*rial , 

T^ie idve Line(s) (S) Option provides a capability for savim up to 
j1 bnes 0^* laterial in a "Save" buffer. The saved material can then be 
i'sserttu at my ;ioint in any textual content frame in any CAI nodule. 
T-tis IS use-^ul m cditinq .naterial, conyinn pages of frames without 
rcty.iiri'i m'i lovinq material to other modules. Once an autnor has saved 
one or iort- lines, they can be retrit-ved at any time until the Save 

T.-,M Fetun Saved Line(s) (F) Option provides the means by which 
a tern 1 is r.nrievmJ fron the Save buffer. If, when the material is 
fetcned to be inserted on a page already containing text, the inserted 
(••iterial will result in a total, of more than 21 lines, the author is 
warnevi uf tnis fact and <;iven the o;»tion of cancel inq the Fetch corrgaand 
or allov^ir,- tne Uditor to reformat the text to overflow onto subsequent 
,hj jes of tn«' ffa ie. 

Trie u>i>tj Save i^uffer (t) Uption simply erases tne contents of the 
^avo uufftjr ;jrior to savinq new material. 

rn.« ^raw/^rase !;ox(es) io) Option allows the author to <jraph- 
icdll,' sjrrojnd one or uorc lines of material for erpnasis. An example 
is 1ft Ti iyre j. :>oxes cau be virawfi arouna any sinjle lin<; or 

irouv iif lines. Trie autnor enters the nunber of tne, first ano last 
]if>(5 t*^ i»t. sjrrOijnded and tfjc horizontal wiiith of tne box i*", aeteriained 
1/ tn<.' ,>osUion of tm> first and last characters in the lines. Up to 
sixteen i*oxt*s can ue Uefined in a sin^jle frar*.e. 



rWg $; Tl; PfCE I or I REPLACE HOOC 2 Lin^m 

1 This 19 an example of a niiap«U«d u^rd 

> This IS an example of a missp 



^ that need editing* 

*DPTI:mS: Enter '^e^t material r**vision. 

Press NEXT to end each line, 

COPY keys copy line to be modified. 

Press 6ftCK when finished* 
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^iqure 7. Author*' t Editor Text Frame Display In 

Replace f^cle. 
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FUSE CLASSlrtCAtlON ' 

6cxnb fuses «r« classified in T>REE WYS--by 
POSITION, by FUNCTION, snd by ^ETH0O OF rt^ING. 



-►'^ By position relates to the POSITION the 
fuse will occupy m the bon4>. 



<=> fuse can be of three position types 



o a NOSE fuse if it is used 
m the FRONT of the bomb. 

o a TAIL fuse if it is used 
in the P£,m o'' the boiTt>» 

o rtJLTIPOSITIONAL if it can 
be used m EITHER nose or 
tail POSITION. 



OPTIONS; I, insert line(s) R, r^lace line 

D. delete line(s) B« draw/erase box (es) 
S. save 1 1 nets) F, fetch saved line(s) 

E, empty save buffer 

d I splay /change next /previous page of frame 
BflCK, return to frame selection 
^CXT, display/change text for next frame 
ENTER Choice > 



Fiqure Authorin-i tditor Text Frame Display 
Showing Use of Boxes for Lmphasls. 
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The final three options pertain to the mechanics of page and frame 
selection. To move forward to the next page of the current frame or back 
to the previous paqe, the author enters or respectively. Press- 
inq the NtXT key routes the author to the first paqe of the next frame in 
the sequence. The BACK key returns the author to the page, of the Frame 
List display listing the current frame. 

Althouqh the Authoring Editor does not currently provide any exten- 
sive 'jra^^nics capability, the author can, with some ingenuity, create 
simp, • qrapnics figures by using the standard keyboard characters and 
any "uarecoynized" character, which creates a so»1d, character-sized 
rectangle. An exanole Is provided In Figure 9. Note that the display 
is snown In Ins:»Qct Only node with an abbrevlateo list of author options. 

Question Generation . Multiple choice and constructed response 
question frarr.es provide the author's primary means of interacting with 
and evdluatinn the student's understanding of the material being pre- 
sentetl. AS structured by the Authoring Editor, a multiple choice 
question consists of the question stem and two to five alternatives. 
After seluctini this frame type (QK) from the Frame Types menu, the 
autnor is S!.o -n a template display and prompted to enter the question 
stem. Tne tttx: of the stem is displayed as It is entered and a double 
pr^'ss of tt«e :ilxT Key notifies the Editor that the st:'n has been con- 
plecea. Tof? author is then prompted to enter the first alternative. 
A double M.<T indicates the end of each alternative and elicits a prompt 
for tnu next alternative. This process continues until the author in- 
dicates t'lat all alternativels have been entered, by pressing BACK or until 
ihf :«xir;urs of five alternatives nas been entered. An example of a 
partially con-leted r.jultlple choice question is shown in Figure 10. 

irice tnt' question stem and alternatives have been entered, the 
dutnor is prompted to specify the correct alternative or alternatives. 
At leai>t one correct answer must be defined before the author is allowed 
to exit the question fram, . In addition, the author Is prompted to 
S{»ecify tne naximum number of attempts allowed In answering the question. 
Tht autnor nay select a number or indicate that the default value, 
currently "three, be used. 

A constructed response question consists of the question stem and 
one to nine anticipated responses. When this frame type (QC) is 
selected, tne procedure followed is essentially the same as for a 
Mujti pie choice question. The author is prompted to enter the question 
sten and successive anticipated responses up to a maximum of nine. The 
correct response(s) and maximum number of allowable attempts must then 
be specified. A completed constructed response question is Illustrated 
in Fiqure 11. Tne asterisks to the left of four of the alternatives in- 
dicate that any of these responses will be accepted as being correct. 

Currently, the AIS CAI software Supports only exact matches on 
constructed response questions. That is, other than variations between 
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G: dtsplay next. 'previous page of frame 

S, save line(s) E, empty save buffer 
BACK, return to frame selection 
NEXT, display text for next frame 

ENTER Choice > 



fiquftt 9. Authoring Editor Text Frame Display in Inspect 
Only Mode Showinfj Use of Siiaple Graphics. 
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S; qrn INSERT hOOC y.n 

TURK alternatives* 



I. This IS alternative one* 



2* This IS alternative two. 



3- >This IS alternative three. 



OPTIONS: Enter nultiple choice question alternatives. 
Press NEXT to end each line. 

line uiith no material (just a NEXT key) will allow 
qMest i on ,a 1 1 ernat i ve ent r i es * 
COPY keys copy previous line. 
Press BACK when finished* 



Flqure 10^ Authoring Editor Display of Partially 
Completed Multiple Choice Question. 
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OPTIONS: I. insert !ine R, r«pUce line 

D, cl«ict« line C, specify correct enawers 
r. diaplayy^change f e e d b ac k mee—ge e 
P, displey/chence proinpt meeeecee 
fiflO<, return to frame selection 
^CXT, diepley/chence te^tt for next freme 
ENTER Choice > 



Firjure n. Authorinq Editor Display of Completed 
Constructed Response Question. 



upper and lower case, the student's response must exactly match one of 
the author's anticipated reisponses. Otherwise, the student's answer 
is treated as an unanticipated response. Work Is currently In progress 
to support recognition of key words and strings a^d misspelled words 
which phonetically match an anticipated response. 

To provide students with substantive Information about their re- 
sponses, provision has been made for author entry of feedback and prompt 
statements for both multiple choice and constructed response questions 
(Options F and P in Figure 11). The Intent of a feedback message Is to 
tell the student that his or her answer was correct or. If It was In- 
correct, why it was wrong. For each altfernatlve or anticipated response 
defined, the author can write a feedback statement which will be pre- 
sented whenever a student selects that alternative. For constructed re- 
sponse questions, a feedback message can also defined for the 
occurrence of unanticipated responses. If the \i:*Mr does not define 
a feedback message for an alternattve, a standard statement, matched to 
whether the response is correct or incorrect, will be selected at random 
and presented. The author can suppress a feedback message by entering 
one or more blank characters. 

The intent of a prompt message is to guide the student toward the 
correct answtr. Whereas feedback messages are alternative-specific, 
pronpts are selected and presented as a function of the number of the 
stuaent's attempt at answering the question. That is, prompt message 
nunber one will be presented following the student's first incorrect 
attempt, prompt two following the second incorrect attempt, etc. Prompts 
are not presented foUowinq a correct response but are otherwise inde- 
peniient of the specific alternative chosen. 

Both feedback and prompt messages may consist of up to three lines 
of 60 characters each. Depending on the author's actions, one or both 
messages will be displayed at the bottom of the screen following a 
student's response. The prompt statement ininedlately follows the feed- 
back statement, giving the appearance of f single .Informative message. 
This concatenation of feedback and prompt: provides the author with a 
powerful tool for responding appropriately to students' errors. 

Question generation mode provides the author vffth some of the text 
editing features (insert, replace an - del>ete} described under Text 
Generation, The use of these features is, however, somewhat restricted 
due to the structured nature of question, entry. The save buffer and box 
options are not available. 

uefinition of Branching Logic * In l|[>repar1ng a CAI module, defini- 
tion of effective branching logic may require the greatest thought and 
be the most difficult part of the procesfs for an author to understand. 
The effectiveness of a lesson, however, jcan hinge on how well this 
feature is used. Therefore, particular, attention was given to 
facilitating this aspect of the author i jig process. 
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Wurnn thg AIS CAI schene, all branching decisions are associated 
with maivldual frames and caji be evaluated at three different points 
in the presentation of the frane» Pre-fratne toqic Is evaluated at the 
point at which the Presentation Proqran first encounters the frame, 
before it is displayed. Re«>oonse logic, which can only be defined for 
a question trame, is evaluated as the student answers the question with 
pe branch bsinq taken if a particular response Is made. After-frame 
loqic is evaluated after the student has indicated a readiness to advance 
to the next frame. A branch can be made to any frame within the objec- 
tive wnicn nas oeen defined in the Frame List alsplay. 

Currently, four types of conditions can be evaluated by Pre- or 
After-fran»e branchinq logic: (a) Given a set of question frames, take 
the oranch if at least a specified number of these questions were an- 
swered correctly, (b) Given a set of question frames, branch if at 
least a si>ec1fied number of these questions were answered Incorrectly, 
(c) Jranch if at least a specified number of a given set of frames (of 
any tyoe) nave been presented, (d) Branch if at least a specified number 
cf a 'jiven set of frames have not been presented. In addition, uncondi- 
tional brancnes can be defined where the branch will always be taken If 
the branching point is reached. As mentioned above. Response logic pre- 
sents a different situation in which the branch Is taken If a particular 
response is made. 



Mny number of branch! m lo>iic Instructions, of any or all of the 

tyi.es, can be entered anainst a single frame. The only restriction 
« total storage space required for all branching Instructions within 
ctive. At least 300 instructions can be defined for a single ob- 



three tyi'es, 
is the 
an otgect 
iective. 



lirdncninq instructions are evaluated in the order In which they are 
encountered. Thus, whether or not a subsequent Instruction will be 
evaluated is dependent on the result of evaluating the current instruc- 
tion. If none of the conditions of Pre- frame logic are met (or If no 
Pre- franie logic was defined), the frame will be presented. If none of 
tne conuitions of Response or After-frame logic are met (or If no such 
logic v^as defined), the next frame In the sequence will be presented* 

The process by which an author defines branching logic for a frame 
is .Illustrated by Figures 12 through 2U. Like most aspects of an 
autnor's work within an objective, the process begins on the Frame List 
display where the author s^.ects the Branching Logic Option (B) and 
enters the number of the frame for which logic is to be defined. This 
results in the branching Logic display shown In Figure 12. %ti ihs 
example shown, no logic has yet been defined for the frame. The authdr 
selects the action to be taken from the Options list—in this case, to 
Insert Frame Loqic (I). The author Is, then asked whether Pre- frame. 
Response, or After-frame logic is. to be defined and a brief description 
of each of the three types Is provided. Assume that After-frame logic 
Is to be defined, for which the Author enters an *A," This results In 
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dmnchint to^ic for Ob)#ctiv€ Is frwm 24 
Multiple Choic* Question 6 

Pr«-rr#me Logic - none 



Response Logic ^ none 



After -Frame Logic - none 



/ 



OPTIONS: Frane mmber, dtap'ay/chartgc frame logic / 
NEXT, display/char)g« next frame logic / 
I, insert frame logic D, delete frame. 4ogic 
B^^K, return to frame list page / 

CNTER choice > i / 



Fifjure 12. 
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Ent«- the frane nuirb«r to b« pr«««nt«d if the logic 
conditions specified are met. NEXT wi ij present the 
current frame. 
ENTER choice > 28 



figure li. Authoring tditor Display for Selecting To-be-branched' 
to frame for Pre- or After- frame Branching Logic. 
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The log^c to «v«lu«^:«d is to b« b«Md on one 

tK« following conditions: 

C. if a set of frames (questions) is correct 

I, if a set of frames (questions) is incorrect 
I f a set of frames has been presented 
I f a set of frames has not been presented 

NEXT, no conditions! an unconditional branch is to occur 

OfT£P choice > c 



Mqure Authoring Lditor Display for Selecting Class 

of Pre* or After-frame Branching Conditions* 

ERIC 52 
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frim Li>t for Cntrv of Lopic Coi>dition Correct 



I Di • oris 

5 SI 23 Qri7 

5 Ti 25 gri9 

' W 2/ T9 

8 ■'■2 -» 28 T!« 

^ ^3 126 E^C OBJ 

El 

f^dfnes which f^st m«et the conditions specified will b« 
denoted by an • to the left of the fr«me number. To 
change a designation, reenter the frame nuf»t>er. To 
cienote additional frames, enter the frames nuitt>ers and 
«n * will ^ppe^r. Press NtXT when finished. 
ENTER frame number > 23 



ricjure lu. Authoring Editor Display for Selectinq Question 
Frai.ies to be Evaluated for Correctness for a 
Pre- or After- frawe Uranch, 
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Enter the number of frames which muat meet the 
condition specified in order to present the desired 
frame. 

ENTEP choice > 4 



Fiqure 16. Authoring Editor Display for Indicatinq 
Mumber of Correctly Answered Questions 
to Meet Pre- or After-frame Branching Condition 



Branching Logic for Obj«ctiv« i, Frame 24 
^Wtltiple Choice ^u— tion e 

Pre-Frame Logic - none 



Resf^onse Logic - none 



After -Frame Logic 
I Ck> to Tif If all of the following are correct- 



f^rame number, d i sp ' ay/change frame logic 

NEXT, d I splay /change next frame logic 
I. insert frame logic D. delete frame logic 
GfCK, return to frame list paff« 
eNTEP choice > 



Figure 17. Authorinq Editor display Showtng Completed 
After- frame Branching Instruction. 
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Er>tet" tKe frane to b« presen^red correspond i rig to the 
alternative choaen* If no response logic is to be used 
for the alternative, then press NEXT. 
ENTEP fraine number for alternative 2 or NEXT ^ 26 



r^jure Id, Authoring Editor Display for Definimi Response 

Contingent Branching Logic* 

ERIC 



Branching Logic for Objective 2. Frame 24 
^Kjiltipie Choice Question e 



Pre-Frame Logic - . none 



Response Logic 
I Go tc when alternative l is chosen, 
c Go to t2 when alternative 2 is chosen. 



ftfter-Frame Logic 
^o to Tia if all of the following are correct 



C-P^ IONS: Frane number, disp' ay /change frame log) 
NCXT. di splay /change ne-xt frame logic 
I. insert frame logic 0. delete frame logic 
return to frame list page 

CN^ep choice % 



Inure 1 Agthorinq Editor Display Showing Completed Respon 
Continqent and After-frame Branchinn Instructions. 



Branching Logic for Objective 2» Frame IS 
Constructed Response Question 1 



Pre -Frame Logic 

1 Present this frame if all of the following are not presentee 
T4,T5 

2 Go to T6 I f al I of the foUooung are correct: 

3 Go to T6 J f I or more of the foHowir^ are presented: 

Re^>onse Lc^ic 
: So El u*hen alternative I is chosen* 
«r Ik t: £Z uthen alternative 2 is chosen* 



After -Frame Logic 
t: E3 if 3 or more of the following are incorrect: 



OPTr>iS: Frane number, disp' ay/ch«ng« frame logic 
NEXT, display/change ne-xt frame logic 
I, insert frame logic D, delete frame logic 
BACK, return to frame 1 ist |>age 

EN^ER choice > 



ri'iure 20. Authoring Editor Uisnlay Showing Three Classes 

of Branching Instructions, 



the display of the list (shown in Flqure 13) of all of the frames which 
Tl^u'V.:??'^''.' the objective. The cirrent fraL (k^^ mbe^ 24 
)L J^'^l indicated by being boxed In. The author is to enter 
the number of the frame to which the branch is to be made f the lod^c 
conditions are met-frame number 28 In this example. ^ 

to-be-branched-to frame has been selected, the display 
changes to that snown in Figure 14. The arrow pointing to frame number 

indicates the frame to which the branch will be made if the conditions 
are met. The author is now requested to select the class of condit ons 
to be evaluated. Assume that the author wishes ' ) branch if sol^ nimbi r 
of questions was answered correctly and, therefore, enters "c!" The 
author IS then requested (Figure 15} to enter the numbers of the frames 

he l?rif%'hf'f '-'^r 'J"^'^^ ^" asterfsris Shown to 

the left of the frame in the list. When this step is completed (Figure 
I0) the dutnor is asked to enter the minimum number of qLt ions which 
fn r ^%^"^^^^-^^,^o-^ectly for the branch to be taken-.in this case, all 

'he ionlc Lf?n^;^'Jr l^^ r''"^ '"r^ ^^'^ ^^"^'^ instruction and 
a.e lone defined for the frame thus far is summarized as shown in Figure 

on tn fhp"!! i'^'r'^* "^^^ ^"^^"^ ^''^^^^^ ^^^I'c instruction, go 

on to the next frame, or return to the Frame List page. 

Assu'^e that tne author wishes to enter a Response Ionic instruction 
^^er options I (Insert logic) and R (for Response logic) have been 
selected tne text of the question is displayed with the correct 
answers) indicated to refresh the author's memory as to its content 
..ext. th^^ autnor is shown a variation of the Branching Logic Frame List 
as lustrated in Figure 13. For each alternative, the author is to 

orancM I Tf.T ^'' lu^ "^^^ °' '^''P^^ ^''^^^ ^he NEXT key if no 
orancn is to be tdKen when that alternative is selected. As they are 
enteren. tne numbers of the alternatives are displayed to the left of 

i?L?^7f ;r''"'/T^' ^? '^'^ this case, the author has in- 

aicatcd a brancn to frame if alternative 1 is selected and is in the 
process of indicating a branch to E3 if alternative 2 is sllecled Wer 
a 1 alternatives nave been considered, the logic which has been defined 

L"?LicaL1"'or1nis fra'n:!'^^^ ' After-frame logic will 

i»oiie of tn- nore extensive branching possibilities are illustrated 



uy f-i>3ure c>). 
'■'t' CAI Presentation Programs 



Wnile Mutnorinq Editor iiroviues the vehicle by which CAI materials 
are uoveloned. tneru must also be software to support presentation of 
t'u>s.^ nateriaU to students. CAI Presentation is accomplished througn 
.h. u<.*. of a lem-ral pronran structure ami stit of support routines 
•rivor. tnf> Cm! lodule descri r-tion . decision Ionic, and text records 
cn ate.. I V an author using the Authoring Editor. Through the use of 
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cms fjfneral izt^o protjran structure afid a table-driven approach, a wide 
ranne of corn uttr-asslsted tutorial and drill and practice Instruction 
can be ,>resented witn minimal pronraminq .effort. To date, three 
different CAI Presentation Proqrams have been developed from the basic 
skelptdl structure and sup,>oct routines. One, the first developed, pre- 
sents stciridard nodules in support of lessons assigned on the student's 
first .>ass tnrounn a olocK, The second supports block review nodules, 
dssi'jned just »>rior to a student's- block test, which present material for 
Just ti»osc ot^jectives which tne stuaent failed while studyinq in the 
clock. The third suoports block reniediatlon modules, assiqiied after a 
blocK t!^st failure, and presents material for just those objectives that 
tne stuuent failoG on tne test. 

Tnc i)asK Skeleton and support routines were written durinq the 
ot*volo.»'ier,t of tne first-pass moaule program. This proqran required 
ti»«, nist extensive design and developinent since it was to forn the 
t.asis -iT suL»s^' ,^ent rroqranis. The block review and reniedlatlon pro- 

werv^ t* created by sli mtly modifyinn the nain loon code, 
^HV'Mv . tine 'or variations to the uasic program nas proven to be 
.i' l ul Can ;.e done Py entry level proqranmers. It should be noted 
t'idt tne first-Kass podule presentation program Is sufficiently general, 
to 'j,a,h'<iri vne presentation of all sucn CAI modules written for any mIS 
.Cojtso. Si'iiUrly, tne review and reniedlatlon programs are general 
er..,...; . to :'a'-.v:l>i tne presentation of any co:jnitive objectives for AIS 
r;},k review ,it»u rerneoiation. 

I", a^jni"-'^ to supjortinn CAI presentation for stuuent study, the 
-i-i r . r.,.S''nt.U] in programs can ae used by subject matter experts and 
ii.!.tr iCtiot>v\l Jeb.i':ners »fha wish to view the module froni the student's 
•/t.-rs. • „ti Vfc. 

T ie followin'} section describes the typical sequence of events which 
occ jir^s w i.?n 1 student interacts with a CAI module of the type developed*, 
specifically, assinnmeat to a first-pass module. The subsequent section 
describes tne use of the presentation program by an author or reviewer. 
Tnt student performance data collection actions which take place during 
t(»e presentatior. of a CAI module are mentioned only briefly. A full 
ui'scripti » of the data acquisition and analysis process is presented 
if! f^.e Subsequent section. 

Student CAI Scenario . As was discussed in the Introduction to this 
re,jar't, wnen tne AIS makes a lesson assignment, it determines whether 
Uiere are two or more alternative nodules, including any CAI modules, 
available for teaching that lesson. If so, there are a variety of 
decision. rules for determining which module should be assigned. As a 
general rale, CAI modules are assigned to that proportion of the students 
for whom they are considered to be the most appropriate, assuming that 
the required instructional resources (in this case, an interactive 
terminal) are available. 
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When assigned a CAI module, the student signs onto any available 
!2^*r^^Jlc® terminal by typing in his or her student account number, 
if the AIS does not recognize the number, the user will be told that the 
account number is not registered and will be logged off. If the number 
u recognized, the student's Student Dat^ Profile record is checked to 
determine whether the student's current assignment is indeed a CAI 
Miodule, If not, the student Is so informed and logged off. 

If the student is assigned a CAI module, the particular module is 
determined from the Student Data Profile record. The Presentation Pro- 
gram then reads the CAI nodule description record, created by the Author- 
ing Editor, to determine whether or not student response and decision 
point data are to be recorded and whether student comments are to be 
elicited. If so. data recording is activated for those classes of data 
which are to be collected. 

The Presentation Proqram beqins by presenting the material in 
Objective J, containing at least an overview of the lesson and a required 
materials list, ana then begins the instruction contained in the first 
numbered objective in the series. Frame descriptions, branching logic, 
and text records, all created by the Authoring Editor, determine the 
sequence and content of the presentation. While authors can define a 
variety of different instructional strategies, a sin-jle approach will be 
JescnLcd nere tor explanatory purooses. First, the statement of the 
objective IS i-resentod and briefly elaborated. This could be followed 
.>y one or fiore pages of text and a set of practice questions. The number 
of practice questions could, and should, be a function of the student's 
perfor.dnce on ;,r1or questions. Additional franes, elaborating specific 
nroblM-^ areas will De presented, as necessary, on tne basis of the 
stviuei)C*> res..onsjs. A typical Dane fron a text frame, as seen by th^ 
itjfjorit. is : restirjteu in Finure u'l . 



In 



^;.t an/ ,.oit,t. in the objective, the student Ray opt to review the 
altrial tnat nas already been presented by pressim the BACK key. I 
rwiew iodL<, text is displayed in the normal manner and questions are 
•iis.'laye.! witn tne student's answers indical-'d. 

'vjestinns and tne feedback and rronpts followim incorrect responses 
tf»r a critic.=j] nart of the instructional process. For botn constructed 
res.ionse and nuUinle choice questions, students are required to continue 
answerinn until correct or until reachinq the specified maximum number 
0^ attenpts. For .nultiple choice questions, the student's last response 
is indicated by an arrow pointing to the alternative selected while 
.>rior respor ses are indicated by asterisks. Fiaure 22 presents an 
ojtarirle in wnich the student is about to make a third attempt to answer 
cne luestion. Note the author-supplied feedback (first three lines) and 
;-ror:^t (last tnree lines) at the bottom of the display. On constructed 
resiorise questions, the student's prior incorrect responses are listed 
ocr.eatn t>«e question stem. Fiqure ^3 presents an example in which the 
stud'.'f.t '^•.aae two incorrect resr-onses followed by tne right ansv/er, nas 



ERIC 



57 

61 



Object iv 1 FRRTC 4 BfCK to review 



All missiles have at least three distinct 
sections. Look at Figure I. The Guidance and 
Control Unit, sometimes called the G and C Unit, 

uiar head and the rocket motor are the componant 
Da«'ts of a missile. 

M^lDftKE and CONTROL I UFlRHEf^O 1 ROCKET nOTQP | 

t 

TYPicftL missile: 

(three sections) 
°ress ^CXT to go on or C to comment on this tevt . 



Fiqure 21. Presentation Proqram Display of a 

Text Frame. 

b8 ff*? 



Obi<ct>v I nWC 58 Pr—m bf<K to r<vii>i 



Detonation of th« AIM- 4 warhead wi U occur when 

1. either the contact fare or proximity fuxe 
contacts the target. 

• the inpact fuze contacts the target. 

3, the missile is 265 to 325 feet from the 
launching aircraft. 

the triggering areas ^re broken by or ;ome 
tn contact with the target. 



Tell me would you want to be 325 feet from an 
missile when it went off*? Neither woui-* a pilot, that 
kind of stuff will mess up your hair. 
Okay, you re my buddy so I ' 11 give you a hint. 

IT S THE TRIGGREING ftREAS. 
Good hint huh' I got all the answers. 



TRY fiCAJH 



/ 



/ 



/ 



Tiyure IL. Presentdtion Program Display of Multiple Choice 
Question Showinf) Feeubdck and Second Attempt Prompt. 

ERIC 
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In Mh»t block of th« flTTO rorm 3Si should th« 
^«<l«r«l Supply Cl«Mific*tion bm r«eordcd? 

Enter • short r«spons« of 21 chsr«^«r« or 1 
«nd pr«M fCXT. 



1. 
3. 



$ 
7 

It 



GOT IT. Gl«ci you finally u)ok« up 



1 1 i 



Press fCXT or C to conmsnt on this question. 



Figure 23. Presentation Program Display of Constructed Response 
Question Answered Correctly on the Third Attempt. 

St 



received a standard feedback nessano for a correct resnonse and Is 
ready to continue. 

Over tuo course of a lonn tiodule, tho author 'nay want to encouraoe 
the student to take one or more oreaksj to exit the nodule ami leave 
the tennnal for a short rest period. Other forms of module inter- 
ruptions can occur due to computer failure, end of shift* or breaks for 
^leals. In eacn case, tne student can loq off or. if tne Presentation 
ProfTd.i vioes not receive a keypress for d specified period' (currently 
1^ nir.utos). u IS assutaou that tho student has left the terininal and, 
dtten cniJla^inq an inquiry as to whether anyone is there, tne student 
IS loyqed off autonatically. If a module is interrupted for any reason, 
whe Presentation Prooram autoinatically restarts the nodule at the frac.ie 
on wnich tno interruption occurred wnen the student loqs back onto the 
term rial, 

after d periou of instruction and practice, an objective typically 
enus v.iin a series of test questions. Given tne criterion tnat a 
v.virtairi huoiber of tne questions nust be answered correctly, the autnor- 
uefined hrancnln^ woulu nonaally route tne student to tne end of tne 
oljective as soon as the criterion nas been net. If, on tne other nanu, 
Lne SLudei.t.s ^ erforr..ance is below criterion, the student would nonially 
not exit tne objective until trbublesone j)oints have been reviewed and 
retesteu witn additional test itens. 

I'pon exitinn the objective, the stuaent will pass through a frame 
aoamst wnicn either an Objective Passed or Objective Failed Flag has 
been set. If tne objective is considered to be prerequisite to a sub- 
sequent objective, a Lesson Failed Flaq would normally be set against 
tne same frane as the Objective Failed Flan. The student will continue 
t'.rou^f. the objectives, in sequence, until all of the objectives have 
been [presented or until the Presentation Pronran encounters a Lesson 
Passed or Lesson Failed Ua'j. 

When a Lesson Passed or Lesson Failed Flag is encountered, the 
Presentation l>rogrdrrt qenerates a module test forra containlnq a list of 
any objectives failed and a lesson passed or lesson failed desiqnator. 
The Proqraii passes this fom to the nain AIS panagenent program, the 
Adaptive Model. Tne Adaptive Hodel records tne student's perfomance 
on tne lesson and generates thp student's next assignnent. This assign- 
r.ient is disislayeo on the terminal for the student to copy. The student 
IS then logqed off the terminal and the module's Instructional resource 
(the terminal) is returned to the resource pool for reassignment. The 
student can then obtain a hard copy printout of the next assignnent by 
subjuittinn a request form to a management terminal. 

Autnor/kevlewer ftode . Access to the CAI Presentation Program is not 
limited to students. Lesson authors and reviewers may wish to run the 
Presentation Prograi.i to verify accuracy of module content and to view 
the module *^rom the student's perspective. Author and reviewer access 
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to the Presentation Programs Is. however, handled by standard AIS pro- 
gran access methods rather than being under the control of the Adaptive 
i'todel. The CAI Presentation Program differentiates between students and 
registered authors and reviewers. The author/reviewer does not have a 
Student Jata Profile record contalninq a current ass1anment» and there- 
fore IS required to enter the identifier of the CAI module to be pre- 
sented* 

Havinr dccesseU a particular nodule* the author/ reviewer may» with 
a few exceptions, take the module like a student. Unlike a student, an 
duthor/rtjviewer may override the frane control logic of the Presentation 
Proqran and request the presentation of any franie within an objective 
ttiroufj.i tne use of a special function key. In addition, the autnor/ 
revitiwcr lay always enter conwents about the rnaterial that is beinn 
prestr.teu. Student consents are only elicited and accepted If tne" 
Student cot.r.ent riouule Flaq nas been set to True. Completion of a Module 
in autJor/ rev lower -node does not result in sub»nission of a lesson com- 
;^UrtcU or railed forn to the Adaptive .Model. In all other respects* the 
«;st:r's intordctiup with the Presentation Program is identical to that of 
d student. 

btudeiit Perron idrice data Acquisition and Analysis 

I'lc t >I Jatd Mc-iuisition and Analysis systen consists of four 'jajor 
CO- , oticMts : aata rtcordin;! routines in the Presentation Proorans; a 
...ita cmI l..c:iv)r; 'Vn -raM which ;iovc:S Student perfornance data fron disk 
u ti.'s;; d -atu -Analysis ke»iort Pronran uaich nenerates three different 
t/,'.^s rc.orts; an.i a P.eport Subt-iittal Pro'iran which facilitates users' 
re4j.;<»t*i *'cr s -ecific re^'orts. 

T'tu- s:.i,j^-f.:. . erfopiiance Jata collection and andlys-«s process begins 
.^U' lat.i rec'^r.tit" rout1r»es enbedded in the Presentation Pronrans. 
Imvs- n itir».s u. erate interactively with data beinn recorded directly 
unto :u. . Tr... mfoniation recorJed is senreqated into two files: 
*'eb;or.s- Point data, which are recorded followinn every frane; and 
jecisiun Poif»t data, recorded only at specified Decision Points, 
•''eriovnuil ly» cie data are duswjed fron disk to tape by the batch CAI 
.-aid Loll^^ctioii Pro'iran, The CAI Response and decision Point history 
ta.^es ,.roviue t^j iTinary data source for the CAI Data Analysis Report 
Prtuira!.. T.ir; \u, ort Pronran can, nowever, also retrieve recent data 
stores ir f e Jisk files for amplications requiriartn small, current stu- 
deni sa :i» les. Tnere are both uacknround (CAr-'lL) and batch {PASCAL) 
ycrsitnii of ^ata Analysis Report Pronram. The forner is used to 
access data un Jlsk while the latter accesses only the tape data. A 
single "IL proaran provides for setup and subnittal of both the back- 
'fiCoufiO anu ;*atcH versions of tne Report Proqran. 

Tr;e jata collection process and the reports available are discussed 
in :riate): detail in the followinn Subsections. Although tne pi^ocess 
c'K.loye-i is sot ewhat different than for ,jerfornance data collection, tne 
">t)ra':o luc retrieval of student and reviewer connents are also described 



udtg v^oUectlon anu Storage . UiH^t»»er student ;)erforMa»ice data atiU 
s'tt'.ts are collected Uurin-^ irvj ,>resJintat1on of a Ul nodulo is depen- 
uef«t on «netr«er tno 'louulu author nas set the approijrlate data collection 
Hu'-'s/or t^\n wuule via tne AutHorim tditor. That is, for uata within 
ont or tnf three catenories (".es^.onse, Jecisioa Point, anu Co'.rients) to 

saveu. tn^- a-jlhor .lust nave set the i)ata Collection Fla-j for tnat 
catconry to True. Tnis »milosot)hv of li-»1ted data collection was adopted 
tn av'M.: 'iKjm»ration aiid storaie of tno i^ienSe arnounts of data wnicn 
woul.. ot'«ervnse occur. The intent is that data be collected for ^oruiative 
anu suinative evaluation nurnoses but not durinn nornal o;»erat1ons except 
for cor.scio'jsly initiated sawplinn^ 

«tesi«'>n5e Jata represents the nost detailed data catenory» The data 
are collecto«j at tne end of each frane presented to the student, renard- 
less of ;o tyrts and include thf followinn i tens J 

1. Tnv student's Student Account Nunber, 

^. Tf.f frane identifier. 

Tne r.uDuer of the student's pass throunn this frane, 

•i. Tne current date. 

~». TMe current tiiTe in .linuttiS after ..lidninht. 

0. T.ie total tine, in seconds, spent on the frai.ie. 

/. T?ie tii.Jc, in Seconds, spent in review mode if review 
was initiated fro?i this frame. 

If thp frane is a question frane, tne followin'j data are also collected: 

v., Tne.nijinber of attempts niade to answer the question. 

The number of the al tentative selected, by atteijpt nuniber 
(where alternative numbers are also assigned to the various 
anticipated responses and the cateqory of unanticipated 
response for onstructed response questions). 

1 ). The response latency, by attempt number. 

11. Tnr number of unanticipated resnonses. 

Ic. The text of up to five unantlcioated responses. 

wecision Point data are collected at the end o*" the module, at the 
enu of each objective, and at the end of each frane aqalnst wnlch a 




Decision Point Flag Has been set. The data collected Include the 
folloMing: 



1 

1 ♦ 


The 


student's Student Account Number. 




ine 


type of decision point (frame, objective or module). 


4* 


Ine 


frdme luentlfler* 


4. 


The 


number of the student's pass through this frame. 


*). 


The 


current date. 


6. 


The 


current time In minutes after midnight. 


7. 


The 


elapsed time since the last decision point. 


8. 


The number of questions presented since the last decision 
point. 



9, The number of questions answered correctly since the last 
decision point. 

10. The number of branching logic decisions processed for the 
current frame; and for each branching logic instruction 
processed: 

11. The branching type (Pre-frame, Response or After- frame) . 

12. The number of the instruction within Its type. 

13. The branch actually taken, if any. 

Both Response Point and Decision Point data are stored on disk as ' 
they are collected. Periodically (e.g., once a wpek), the CAI Data 
Collection Program is run to delete the oldest daca from the disk 
Response and Decision Point Files and transfer the data to the CAI 
History tapes. 

Collection and storage of student (and reviewer) comments Is handled 
somewhat differently. First, of course, the student or reviewer must 
take overt action to enter a comment against a frame as opposcsd to data 
collection being transparent to the user. Up to eight comments are 
stored for each frame in a "circular" file on disk. That Is, when a 
ninth connent Is entered against that frame. Its content replaces that 
of the first (oldest) comnent. The content of the Comment files Is 
never transferred to tape. 

Data Analysis Reports . There are four different CAI Data Analysis 
Reports available to authors and evaluators: the Decision Point Data 
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llnZl' I A ?f P?"^^ Analysis Data Report, the Unanticipated Response 
r.n ll' Listing, In addition, standard AIS CMI reports 

can be used to provide a description of overall module performance, 

^^l ^^^^^^ requested from an interactive terminal. For the 
n^I^n!i i", A? o^*'*^^''"^"^^ reports, requests are submitted via a CAMIL 
^^ZT ^ ^^^^""^^ Proqram) which prompts the user to enter the various 
"T^^J"^^^ P^'^^'^eters. Thus, the user does not need to learn how to 

?nJlude: '"'"''^ '^''^'^ ''^^'^^ '^^''"^^^ parameters 

K Tne module identifier (Course, Course Version, Block. 
Lesson and Module numbers). 

2. Tne type of report. 

3. The date constraints (the period from which data are to 
be drawn). 

4. Tne input medium (disk or tape). 

:>. Tne objectives within the module which are to be reported. 

o. If desired by the user, the numbers of the specific frames 
within each objective to be reported (where a set of 
frdines is defined by the numbers of the first and last 
frames in the set). 

Once tne request parameters have been defined, the report submittal 
proqran -3ts uo the necessary files, submits the job to generate the 
report o.i the central line printer, and gives the user the job number 
wnich will appear on the report printout. 

The uecislon Point Report Is generated from data, stored either on 
duk or tape, in tne Decision Point Data File. This report provides a 
sar.iioary of student • per for.iance within each objective and within those 
intraobjective segments (sets of frames) which the author has defined • 
oy setting jecision Point Flags at the beginning and end of each segment 
M\ exdinp.e page from the Decision Point Report is presented in Figure 
Jh. :,dcn component of the Report contains the number and name of the 
frame representing tnat Decision Point; wnether the data reported per- 
tains to students' first, second, or subsequent passes through that 
point; tne elapsed time, number of questions answered and nuMber answered 
correctly since tne last Decision Point; the branching logic evaluated 
at this point; and the nuinber and percentage cf students takinq each 
branch. " 

nn example page from tne Response Analysis Report, is presented in 
Figure Lacn component of the report is identified by tne number and 
na..ie of tne relevant frame and whether tne data repoEtcitjjer tains to 
students' first, second, etc.. pass through this frame. For franies 
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Figure 25. Example Page from Response Analysis Report. 



other than question frames, only time data are reported. For question 
frames* a matrix format is used to present student performance and re- 
sponse latency data as a function of the response (multiple choice alter- 
native or constructed response) selected on successive attempts. The 
marqins of the matrix provide a sunwary of student performance on the 
question (total percentage correct, percentage correct by attempt, and 
total time to correct response) while the matrix cells provide a more 
detailed picture of how students reacted to the question* 

An example page from the Unanticipated Response Report is presented 
in Figure 26. For each constructed response question, each unique un- 
anticipated response Is listed in order of frequency of occurrence 
together "WTth the number of times which that particular response has 
been entered. 

Consents Listings are requested via the Authoring Editor rather than 
the CAI Reports Program. Frames against which comments have been made 
are indicated on the Ed1tor*s Frame List display for each objective. The 
user can request that comments made on a particular frame be displayed 
at the terminal or that conments on one or more frames be listed on the 
central line printer. The user also nas the option of having the 
conwents purged from the file as they are displayed or listed. 

CAI Material Print Program 

As authors create, review, and revise CAI modules, it Is often use- 
ful to work from a hard copy printout of the module's content In addition 
to, or in place of, the displays provided by the Authoring Editor. There 
are also instances in which hard copy printout is desirable for student 
use, A feature of the Authoring Editor is the capability to request a 
variety of printed listings of CAI module's content and branching logic. 
The Editor queries the author for the desired print options and then 
Initiates a special background (non- interactive, low priority) program 
to produce the printouts. 

Author Listings . In addition to listings of student comments, four 
different types of printer listings are available to authors, ranging 
from sunmary information to detailed listings of frame contents. The 
option of requesting multiple cop.1es is available for each type of 
listing. 

At the most general level, the Module Sunmary Listing provides an 
overview of all of the CAI modules, operational or under development, 
currently defined in the data base. The Information provided for each 
module Includes the module Identifier (Course Number through Module 
Number), module title, author's 10 (Social Security tiumber), and the 
number of objectives defined within the module. 

For a particular module, the Frame Summary Listing provides an over- 
view of'*the content of individual objectives. An example of this listing 
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Is prcsonteu in Fi-jare ..'/. Tne Hifomation provitleO by this listing is 
tssuitti-tny trie sdue ds the Luitor's rrame List uis^ldy . Lacn frame in 
Via oUjective is listed by nainber anJ frdine name. The existence of any 
ordnLiiin'? lo'nc dnu frame Flaqs is noted and the inaximuni nui.iber of 
dtcet:.i>ij alloweu to answer juestions are snown. If a frame is an alias 
(u<',» references anotnor frame)* ti^e referenced frame is identified. 

It.t: ..«j'ot ^'"re jju'ntl y used printout is prubauly ttie f raiae Contents 
uiMUi zn-i ou . h'tt- »jrintuut, h/ fraiie, o^" all text and 'juestion 
.i.it»:riil, , icn a lisUn'j can bo requesteu for aii entire tiodule, an 
iniw»uj.il oLit'ctive, u SiUicificd set of frames or a sin-jle frame, Tne 
t id •.♦.rial contained in a textual content frame* up to the full four panes* 
IS ijrinte-; un a siaqle printer (laqe with appropriate headings. An 
example printout of a Text frame is shown in Fiqure 23. For a Question 
frare* fu.< printout includes the question stem, tne alternatives or anti- 
cipatevi responses with the correct answers denoted, and all author- 
supplied feedback and uronnt m^'ssaqes. An example is presented in 
F inure *, 

f ifially, tne Branching Loqic Listinq, an example of which is snown 
m t i«}ure 3;, proviiles a narrJ copy listing of all of the branching logic 
wrmch has l»een defined for frames within an obiective. The format in 
w.ncn ir.fonidtion is presented is similar to the Ldi tor's branching 
COM It tns.ilays. 

^'^'i"_ted :%iter ials f o r Student Us e. In addition to the various 
ajtnor'^s 1 1 stings T fi^rd ropy printout of a CAI nx)dule's content can oe 
re ,aestej in a foritt^t approi>riate for direct use by students as a 
Mrogra ruM text. Special purpos«- (Documentation and Branching Decision) 
frames are autou-.atically suppressed, and the author can elect to suppress 
any otner frames by setting "Skip this Frame" flags via the tditor. All 
otner frames are printed in the order in wnich they occur in the mouule* 
lirancmng logic is sirtply ignored. 

^or textual content frames, one screen display (a frame page) is 
prin;e;j on eacn page. For question frames, each question is identified 
l>j a number, assigned sequentially. For constructed response questions, 
only tne question stem is printed, wnile the alternatives and their 
numbers are printed for multiple cnoice questions. The answers to all 
questions, identified by number, are listed on a test key page following 
the body of the mnoule. Page numbering is provided automatically. 

Material is currently printed in a double-spaced format. Boxes 
wnicn tne author nas used to emphasize material in the moduH are shown* 
as are any of the simple graphics which the author may have provided, 
for such graphics, a symbol is Substituted for the solid, "unrecog- 
nized cnaracter" symbol typically employed. As an option, the author 
may request tnat the material be printed on special unlined, <i l/^i" by 
11" pai^er. Tne option of printing multiple copies is also available. 
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Figure 27. Example Frame Summary Listing Printout 
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Multiole Choice Qjestion aumoer k 



no rt^r^^e-ilt 



rLtOaACK MESSAGcS 

1 NO, 8oti rocKet and missiles have warheadst 

2 NOt rockets do not have any guidance systen. 

i Way off! The type of orooellent doesn't mattert 



PKOMPT iSSAGcS 

1 There are three oarts oi a missile one of then is 

'^ot used on ^ rocKet* 

i The O'le 5ln<ne thin.i that fleshes a missile a MISSILE 
is toe GjIuANCc SYSTdM. 
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Figure 30. Example Page from Branching Logic Listing. 



b 78 



There are a variety of uses for such printouts. They can be used 
as hard copy backups for students assigned CAI modules In the case of 
computer failure. They are useful to Instructors for answering the 
questions of students assigned CAI modules. Their most important 
function however, may well be as a first step toward the on«line develop- 
ment. evaluation, and revision of materials intended for off-line use» 

Authoring System bocuwentation 

^ CAI Autnoring Editor Us er's Hanual was written to describe the 
concepts supported by the tditor, the mechanics of its use. the CAI 
Presentation Proqrams, the various CAI Reports available and the pro- 
ceaures by whicn they are requested, and the availability and use of the 
different types of author and student printouts. The Manual was actually 
developed on-line as a CAI module (without branching or question frames) 
dnu is available to novice authors via any of the AIS interactive ter- 
niinals Mdrd copy versions of the manual are also produced as needed 
by tne Cm I Material Print Program. 

This approacn has assured rapid and easy revision of the manual to 
provide up-to-date documentation as the Editor and its various support- 
ing programs are expanded and refined. Under a companion. Authoring 
Proceuures contract, the Manual is being expanded to include infoaiation 
on the selection of content for CAI treatment, instructional strategies 
anpropridte to CAI, and guidelines for evaluation and revision of CAI 
modules. 

Mltnou.jn it nad not been implemented at the time this report was 
prepared, it is intended that a second type of Editor documentation will 
be provided, "rtelp" routines will be imbedded in the Editor Itself. 
That is. tnrouyn the use of a special Function Key, the author wilTbe 
aole to access infortaation pertaining to the use of the Editor Options 
wnicn are available to him/her from the currently displayed page of the 
tuitor. Tnese Help routines will access tne information contained in 
appropraite frames of the User's Manual. Thus, hiost required revisions 
to tne documentation will only need to do made in one place. In the 
manual itS'?lf. The Help sequences will then access the updated infor- 
mation automatically. 

Jocur.ientation to support subsequent maintenance and revisions of 
tne Editor itself, tne Presentation Programs and the various supporting 
programs is provided by a Part II. Product Oevelopment Specification. 
Tms Part II Spec was also produced on-line (via a different, less 
structured Editor), is stored on tape and is accessible for the pro- 
duction of nard cooy documentation. 
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In dooition to tne cmI support software effort described Mere, tne 
CAI uevelopoent project also contained a second authoring procedures 
effort. Tne :)jr,ioses of tnis second effort were to (a) define and docu- 
lent d Set of i^-roceUures for C«I materials selection, production, eval- 
uatiotj, dnu inple:jentat1on in the kl^ environment, (b) evaluate the 
utility of tne total CAI dutnorinq system by developing, impleinentiny 
ana tvdijdtHvj a number of CAI mouules in an AIS course, and (c) train a 
SiVdU nuiJbtjr of ATC personnel in tne use of the procedures and support 
software. Tne results of tnis effort are reported in full in Part II of 
ttMs rei*ort (Lewis, Lovelace, Hanany and Judd, in press) and will be 
uutlineu only oriefly nere. 

Tmo dutnorint; procedures effort beqan witn a rough definition of 
tne .iruceUvirj} .louel wnich was then revised and refined on the basis of 
experience, '^oilowinn tne steps outl ined by the procedural wdel , work 
Ln'Nd»> by s^ lectinq cdndidate lessons for the CAI ue»nonstratioiu if tne 
f >ir c>j?-»-,'b, tne wea,JOiii !:echdnic course was considereu tne most 
.'ty i .1', ^ ..vcij^e of its co;.»binition of varied subject natter, lanje 
'.t^.cr.t fUw, d receiit relatively uoor field performance data, 
jwdnuari: -1^ . :! reports were useu to identify six lessons in two ulocks 
of t »e CO irstf wnicn demonstrated unacceptably niqh lesson test failure 
rdtej» n.-'uLi vely ni'jh failure rates on tne ena-of-block tests for tne 
vji.>jeciwe> cuntutnou in tnese lessons, and wnich contained subject natter 
.mliKcly u i!e :?oai fied in the near future. These lessons were targeted 
fur tnree ty,-es of CAI application: (a) six modules to be used as alter- 
native treit-ients on students* first pass througn the block, (b) two 
blac.^ review .woules containing CAI treatment of the objectives from 
tnese lessons, .md (c) two block remediation modules, again containing 
uAI tredtiient of tne objectives fron tne six lessons. 

After exanining the existing materials and tests, it was concluded 
tnat neuner tr.e lesson tests nor those portions of the block tests per- 
taining to tne relevant objectives were adequately valid or reliable for 
evaVuatitig tne effectiveness of the CAI treatments. Consequently, both 
forms of tne two block tests and two forms of each of the six lesson 
tests were substantially revised and expanded to more closely match the 
stated pcrfon.iance requirements of the Specialty Training Standard, Once 
the i^evised tests had been approved by course personnel, they were imple- 
•nented in paper-and-penci 1 form for all students In the course. 

Work then began on developing six CAI modules for administration 
during students' first pass through the block. While existing materials* 
were available in the form of programned text and, in some cases, audio- 
visual iiiodules, the content was substantially revised to more closely 
WW ten the requirements of the Specialty Training Standard. Instructional 
strategies were also, of course, tailored for CAI presentation. 

Work started on three of the modules before the Authoring Editor 
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wds ready, but after It had been designed and the module structure de- 
nned. Tnerefore, the earnest authoring was done on paper display 
forms. When a rudimentary form of the Editor became available, text and 
questions which had been prepared on the forms were input by a secJeta?? 
It was thoumt that some of the authors might prefer the use of forms and 
wish to continue with this approach but. as additional Editor fea?ures 

^ni "J^Ji'J'lf ?^ ^^"'^d convenient to input 

and fomat the materials themselves. All of the work on the last three 

Joun?er.T^ ^ccompli shed this way. Surprisingly few problems were e«! 
countered during module deve1ophient» but many aspects of the Authorlnn 
tUitor were shaped by frequent interactions between the authoring and 
software teams. ^ 

ns soon ds tne modules were completed and their content reviewed 
by ..edpons Hechamc course instructors* they were tried out on a one-on- 
one bdbis witn a smdll number of student volunteers. A nuiaber of minor 
revisions wer« i.idUe on the basis of tnese reviews and tryouts, and tne * 
.«t»dules vn.'re iinpKMjenteu in the course for purposes of formative evalu- 
ation. Following revisions Made on the basis of tnis evaluation, tne 
«K)Uules were reiMplementea for sunnative evaluation. The results of this 

the n^^rntn r^^^ ''^ this report. The objectives from 

tne first-pdss i.iodu es were then copied, revised and snortened. and com- 
Dinea to fonn two block review iwdules and two block remediation 
inouu I es . 

Tuo LmI dutnoring teau consisted of tnree members. All were exoer- 
leuced tecnmcdl training authors (of programmed text and audio-visual 
'wtendls) but none ndd duy prior CAI duthoring experience. In fact 
none ndJ even used CAI ds d student. Only one of the three could be 
considered <\ WedpOfis Mecnanic subject matter expert. 

As is often tne case, the team did not keep accurate records of 
Jevelopjient ti«ies but times can be estinated for the six first-pass 
Modd es. At tne end of tne first b months of the project, the first-pass 
mHij\t'% ndd been revised and implenented for sumi.iative evaluation. The 
tean leader spent relatively little time actually authoring, concentrat- 
ing instedu on defining the procedural iiodel, producing the CAI Autnor's 
(idiHibooK, interfdcing witn the software personnel, and attending to 
ddipinibtrjtive problems. The other two team members were occasionally 
cal leu upon for assistance on other on-going projects. A liberal esti- 
mate of tne total time spent in developing, evaluating, and revising the 
six modules 1!, litdu wanhours. Tnis incluues tine spent in revising the 
oltKK and lesson tests even tnough Wock test revision was technica.lly 
not part of tne CmI effort. ^ 

Mccoruiiig to tne course's Plan, of Instruction (Pdl)...the content 
taujnt by tne lAf modules was equivalent to approximately ^J classroow 
nours. jn tnis basis, CAI development required do mannours per POI nour 
Average student completion time, totalled across the six modules, was 



I "uMjrs. r<iii ro'ijits in an «?itinateU 11.. nwnnours per stuJent 
contaci .lour. r-tost> vdluei are couparaule to uevelopment times for 
;.Tt) ^ra I ted t--*xt idterirtU in tniti envi romient. 

L'sjbiUty of tnc Autnorin*} tultor wd^ also ovdiudted tnrouqr? train- 
iHM tt.reu AT(. instructors in use of the editor. Tnese instructors were 
drdun froo three uifferent courses supported by tlie AIS— Inventory 
Mdnuqenent, Precision Heasuring Lquipment, and Weapons riecham'c. .None 
uf the autnor trainees were coi:iputer progranrjers and none had prior CAI 
development experience. The training period lasted for 3 weeks» during 
wnicn tne trainees came to the contractor's facility for 3 to 4 nours 
eachnorninq. T^e first session was spent in providinu an overview of 
whe Alb CAI s/^»tot. and tne rule of CAI within the AIS and in introducing 
c»>e trainees to tne Authoring Editor and tne CAI Author's Handbook. No 
fon.Jdl trainim took place durinq the subsequent 14 sessions. Using the 
haMdb(»oK dS a reference Manual » each of the trainees used tne Authoring 
Lai tor to develop a CAI module in the area of their own specialty. Con- 
tractor jorsonnel" were available to answer questions and to review and 
uot.nent ofi tne trainees' work. In many cases » author trainees were also 
able to work on tneir i.iodules during tne afternoons while performing 
their nurnal classroon duties. 

Tne autnor trainees asked relatively few questions after the first 
few ^essu'^.. .lost of the suggestions made by contractor personnel per- 
tdinej to toe need for more frequent questions in the modules and in- 
creased individualization through branching. At the end of the 3 week 
;»eriod. eacli trainee nad developed o i.iodule» iiad nad it reviewed by the 
contractor and otner ATC personnel, had run single student tryouts, and 
nad iiiaue Minor revisions on the basis of these reviews and tryouts. The 
consensus of those reviewing the niodules was that they were generally of 
good v^udlity and nacj capitalized fairly well on the capabilities of CAI. 
One of the modules was subsequently implemented in the Weapons Mechanic 
course. Tne ti ne expended by the author trainees on this first module, 
through revision following single student tryouts, was approximately 90 
nours per i»tudent contact hour. 
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V CONCLUSIONS AND RECOMMENDATIONS \ 

*nr^.I?^ Authoring Editor approach to facilitating CAl development 

^ ^^""^ promising. Experience to date, while 
Jdmtttedly limited* has demonstrated that reasonably effective CAI 

^ produced at a very acceptable cost in terms of manhours 
per hour by personnel without prior CAI authoring experience. In 

?,hJ^1*^?J: ^^^"^ '^^C personnel can learn to use the 

kL ^ reasonable period of time without formal training. While it 
^flo!!?.^?^'^'*'^i^^ that theVATC author trainees would have numerous 
suggestions regarding desired changes to the Authoring Editor, this was 

Jhf IT^L . ^ ?f ^^^^ ^" general, quite satisfied with 

the Editor and all expressed an interest In having CAI implemented in 
their respective courses. As it stands, the authoring system appears 
ready for use by ATC instructional development and evaluation personnel. 

In assessing the various features and components of the authoring 
system, the rnajor contributor to simplifying the task and hence reducing 
costs IS probably elimination of any need for the author to work in a 
computer language. All of the progranroing work aas been done beforehand 
and provided in tne form of the Editor and Presentation Programs. Future 
needs for prograntnlng effort will depend on now adequately this software 
meets the requirements of futi ? applications. AHhough it certainly 
cannot be oroven, the authors of this report think that, due to the 
I^Ik ll;^'^"?^^ existing software could serve the needs 

OT the Mis environment for some time to come. Eventually, however, it 
IS anticipated that Jevelopinq author expertise will justify increased 
software capability* 

The second greatest contributor to facilitating the author*s task 
IS probably the extent to which the task is structured by the Editor 
Tne overall structure of the module is determined for the author, units 
within this structure are matched to the requirements of the environment* 
and tne occurrence of critical units is either forced or prompted. 
While the dutnor retains a great deal of flexibility, this flexibility 
IS exercised tnrouqh selection of specific options which provide a 
Jeqree of control over the authoring process, while reminding the author 
of the various courses, of action which may be taken. 

H third major factor in. facilitating authoring is undoubtedly the 
,numan-engineered, computer-aided input, formatting, and editing 
capability provided by the Editor. Other than tne approach to defining 
branching logic, there is little hei4 that Is radical or even novel. Tne 
work involved only the application of existing technology to a particular 
probler.! area. Given the diminishing cost of computer use, there is 
little reason n:>t to provide authors with the benefits of this 
technology. 

At this time, it is difficult to evaluate tht utility of the auto- 
matic, structured, student performance data collection and analysis 
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rojtmes, T-u.- r^.^orti Uiu not t»ecoMe avdUdble until the later stages 
of farwtive evaluation of lut- six f irst»pdSS'iiOclules» Uuriny their 
It iiteJ ueriod of use, tney diu appsdr to be easy to use and interpret. 
Tr)c Me ^oers of t'li^ )articular dJtnorinq teai.) were, how»»ver, dccustoned 
lu UiHoctuH suCM uoitd and uslni the data to improve instructional 
>»'>t-rr ui ^. Tor DrodUcr ai>i)l icatiofi, more extensive pronptin'j and 
•luiuatice in jie of tne. data collection routines might be desirable. 

It ^Mould be noted tn.u the ^iftware developinent effort reported 
here was substantial. Given ti _ extent of tne various functions to be 
su.ijjorted, tne authors of thi . report believe that it would not nave 
been jHjssible to complete the project within the temporal and fiscal con- 
sirdints of t.ie contract if it had not been for the availability of the 
InHil lanvjuaqe, the software developwent capability of the AIS, and the 
well-structureu nature of the Ct-ll systeia in which the CAI was etabodded. 
• »»e rolvitive ease with wnich CA.'ilL code could be produced and debugged 
alluweu the uevelopers to experiment witn a number of different 
a,»,;rodLnes to var,ous problems, obtain user feedback on these different 
.ipprydcnes. and revise the code accordingly. Tne process was essentially 
tudt of toruitive evaluation and revision witnin the domain of numan- 
en.nneert-d software. Tnese satne cnaracteristics of CA-MIL and the AIS 
will fdciiuate any future expansion of tne CAI authoring system. 

wun respect to recounendations for future activities, the most 
critical action that Must be vaken if the effort reported here is to be 
justified, is that the CAI authoring system be used. The results of the 
evaluation described in Part II of tnis re{)ort, although limited, stronn- 
»y indicate tnat. used judiciously. CAI can serve a useful function in 
d tecJHiical training envi rom.ient such as that of the AIS. Further, usa- 
t>i1ity evaluation results reported both nere and in Part II deinonstrate 
t »dt CmI development can be nade cost effective through use of the 
autnoring system. 

It is recognizeu that there are certain inadequacies in the author- 
ing systei-i, pri.idrily in the areas of response processing and autnor- 
generatea nrapnics. These areas provide two of the most likely 
candidates for future software development. 

«s was discussed in Section II of this^report, answer processing 
for constructed response questions iias been a problem area in CAI devel- 
oped for Military technical training. While sopnisticated algorithms 
exist to did in response recognition, the CAI systems employing these 
algorithms nave not made the authoring process sufficiently simple for 
tne algoritnms to be used correctly. It Is recownended, therefore, that 
a response processing dialog be developeo and added to the Authoring 
Iditor wnich would guide the author in defining anticipated responses to 
constructed response questions. The encoding algorithm employed by 
PLATO could be used to recognize misspellings. As currently conceived, 
this diaiog would first prompt the author to enter an anticipated re- 
sponse. The Lditor would break the response down into Its component 




wordi, tind ask tne dutnor to Identify the "iMportant" words In the 
r»iS»ionse» .<oise words such as articles, prepositions, and auxiliary 
verbs would recognijed and the author would be questioned as to their 
importance. ^ext» tne duthor would be asked if word order is important 
and, if so, would be prompted to indicate important word groups and any 
ordonntj restrictions within and among word groups. Finally, the author 
would be pro.apted to enter synonyins for the important words. An approach 
SUCH as tuis would not only simplify the authoring process but would 
nclp tne autnor to identify tne critical aspects of response judging. 
Tnis, in turn, SKOuld promote generation of better anticipated responses 
ana constructed resi>onse questions of njore unifom quality. 

T'u? :>oftwdrL' described in this re:)0rt ..Jdk<iS no i)rovision for autnor 
;»'n«?r4ii uf 'jra.snic displays-. Although sucn displays can be construc- 
:ou t;ir )j in use of t.vi CA-1IL Idnqudge, it is recoi Blended that a graphics 
euicor bt! Jt.'vul )pea wnicn would dllcw the non-prograniming author to 
defir.e drawirvvi in terns of basic qeonietric ele.^nts (e.g., straight 
Hues, circles, and arcs). Such an editor v^ould be relatively- efficient 
-in tcr ,s of storage and redisplay since only tne basic elements of tne 
urawifuj need to do stored and snould provu adequate for preparing simole 
fi'jures and alinost all scher.iatic diagrams. 

If ujro canpiex drawings are required, nowever, it becones inucn 
..Kir*.' d^ffivviU for the author to define the figures in terms of geometric 
•jnt»,)0->. .<-uK' It IS possible to define "freehand" shapes through key- 
board control over a cursor, tn« process is substantially siiiplified 
lurouTi JSt: of a liq»>t ^'en or digitizing device. The plasma display 
teminals j^od by tm> AIS Cinnot support light pen capability. Two 
funs uf jr;itijers are avai 1 db1e--video and tablet. A video digitizer 
jsc> a </ideo cjnora to scan the drawing and convert it into d dot matrix. 
h) jse a tai Ut di'jitizer, tru.* author overlays a special electronic 
lai'lei wK < tM«j drawing to bt; reproduced and tnen uses a stylus to trace 
tnat i»v,«r:mfi of t-ie drawing w'tich to be transmitted to the nost 
coMpuctT and stored. Typically, the tracing process can be eitner con- 

^ tinuous or step wise. Tne ftajor i>robTcM witn a video d^git. "er or 

^OfitiniOjs tracinj on a tablet digitizer is tnat tne drawing is repre- 
S'inte«i \n ter s of uots ratner tnan geometric elements, kedisplay of 
sin;le Jots un a vector tertainal is an extre.iely time-consuming process. 
iiU! di ntueo idtrix is really suitable only for redisplay on a terminal 
wUf* a refr»-*bii i.ie iory wnich can be preloaded with tne graphic. For a 
gra;>»\ics digitizer to \ni feasible for graphics generation using the 
current "i'j t».»r inals, software would be reguirec to trans fom the dot 

rr-- natrix int<i a nai.itjor of basic geometric elements which could oe more 

guicM / redispld/eu. Such jiii/tw3fe"would reguire contour recognition 
routines w;ic!5 wauUHirjImn- trivial to develop. 

- - ^ 

r. Is r-coMiended ti^at tw- use of a tablet digitizer connected 
dir'>cti/ t:> i 'jraptiics eti^tor he investinated. Uit*i s jch an approacn.» 
; t..ie stylus of the digitizer could be used Mke a lignt pen, and the 
— tiblt.'t coul'.l contain a menu of the geojifr'tric snapes recotjnized by the 



3^ 95 



eUKor« To u!.t* sucii a systeia. tne author would select the oeojiwcrk 
elet.Kint ttj rc }rojuco(j frofi tne i.ienu and note the points ref)resentin'; 
Vui II..US or Me oleuent. For example, after the snape was defined 
froM tMe (efia» a line Serpent would be entered as the two end points of 
tne line; a circle would be entered as the center point and a point on 
tne v.irca i.erence. Tnis aooroach could conbine the stronq points of 
ootn t»ie Jiqiti^ier and a ^jraphics editor. 

In a totally aiffereat area, it is sufjnesteu that the utility of 
tne dutnonni) s/sten could be substantially increased through provision 
of .lUUmonal tools for inananinq tne autnorim process. The approach 
envisioned includes capturinq relevant parameters of the development 
process and providing access to this information through suimary displays 
anc rt.';>orts. Only a start has been made in this area, fluch remains that 
can duu 'j'.ojlvj btj done. 

Mi.an,,. it 1^ recoiiinended tnat tne autho:Mng system's capabilities 
tar on- liny ..rouuction of materials intended for off-line use be ex- 
panded, .nis «oulvj be particularly useful if tne additional Management 
tools Mentioned above were also uade available. Software development in 
i.tr. dr.,a mm o mcljue a means of producing scrambled programmed texts 
.roM lesson cdterials and autnor-suppl ied decision logic and a text 
arcniver. natnors could develop materials on-l,ne, use the CAI Presenta- 
tion Progra:. an.j its embedded data collection routines for fonaative 
ev.iljation, idKc needed revisions on-line, print the number of copies 
necJeu, and allow tne i.idterial to be reeved to tape. When additional 
cuMics drc neeuej ur revisions are required, tne author could place an . 
arcnive request to move the lesson material from tape to disk for revis- 
ion or i^rinting. Such an approach would not only facilitate the process 
of development and formative evaluation, it could drastically reduce 
Materials reproduction requirements. Currently, it is coiiinon practice 
to retjuest many :»ore copies of materials tnan are required so as to allow 
for nomal classroom wear and tear. The materials are then often revised 
tie fore many of tne extra copies are ever put to use. The approach 
i,uTjeste'j nere v/ould eliminate the need for these extra copies. 



^2 ^--^6' 



^un<]»;rr..Mi, V. Tf.'dJ.: :>roauction of learner-con trolled courseware- A 
projres^> refort. In L. -inn. :i Ruf1ce» A» Ho»iano (Eds.j. 
Lorputers in tn e Instructional process; Report of an international 
19I122i« Ann Arbor. MI: Extend Publications, 1973. 

.Jdll.tafi. n.^L., .!eLeo» P. J. Main, P. S.. S Gillian, D. C. Evaluation 
of PLmT) IV in vehicle maintenance training . AFI:RL-TR-77-59, 
M.)-A>j2,t23. Lowry AFH. CO: Technical Traininr] Division. Air 
rorce Hun.m Resources Laboratory, November 19771 

I'owsvy. i;. Easy author-entry systems: A review and prototype. 
Internatl. Journal of Man-flachine Studies , 1974, 4, 401-419. 

^^'■"-'^^^^^ H. A. A co-;parison of the TICCIT and PLATO syst ens in a 
miliary settin9. mtc Henort. >}n. Ih. (:nnp..t^.^-Kl^^^ E-iucation 
Kesearcn Laboratory, Univerlsty of Illinois, 1977. 

'^'*»'^"'"» ^'escription of basic author aids in an organized s ystem for 

coftputer-assisted instructioru Cambriuge, MA: Massachusetts 

Institute or fechnology, 1975. 

Kituberlin, J. A. Fifth year status report: Computerized t raining 

systens project, pro ject ABACUS . Report cTD-TR-77>1. Fnyfan^dnn ^ 

Comunicative Technology Directorate, U.S. Army Training 
:»up;)ort Center, August 1977. 

Lewis, L., L&veldce, D. E., Mahany, R. W., 6 Judd, W. A. Computer- 
assisted instruction in tne context of the Advanced InstructicnaV 
System: P art II of II - ilaterials development procedures and 
system evaluation. Lowrv AFfl. CO: Tprhmr^l fr^-^n^^^ nTyj^-^^n^ 
Mir Force Human Resources Laboratory, in press. 

Paloian. A. Y. An interrogative authoring system. Internatl. Journal 
of nan-Machine Studies . ",974, 6, 421-444. ~~ 

Pflasterer, D. C. The CAHIL proqrannim language. SIGPLAM Notices, 1978. 
Vol. No. 11. • 

Scnulz, R. C. Monofornis as authoring aids for the PLATO IV CAI system. 
MumRRO Technical Report 75-5. Alexandria, VA: Human f^esources 
Research Organization, June 1975. 

Sherwood, B. A. The TUTOR language . PLATO Services Organization, 
Computer-based Education Research Laboratory, University of 
Illinois, June 1974. ' " . 

Stetton, K. J., Volk, J., & Bunderson, C. V. Toward a market success 
for CAI: An overview of the TICCIT program. In K. L. Zinn, M. 




Refice, & A. Romano (Eds.), Conputers in the Instructional process: 
Report of an international school . Ann Arbor, Mich.: Extend 
Publications. 1973. 

, K. I . t^equirenents for effective authoring systems and assistance. 
Internatl. Journal of flan-Machine Studies , 1974. 6, 144-352. 



84 



